<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="pretty-atom-feed.xsl" type="text/xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
  <title>Blog Title</title>
  <subtitle>This is a longer description about your blog.</subtitle>
  <link href="https://example.com/feed/feed.xml" rel="self" />
  <link href="https://example.com/" />
  <updated>2026-01-30T23:40:00Z</updated>
  <id>https://example.com/</id>
  <author>
    <name>Your Name</name>
  </author>
  <entry>
    <title>Resize the Kali Linux root filesystem</title>
    <link href="https://example.com/blog/resize-kali-linux-root-filesystem/" />
    <updated>2026-01-30T23:40:00Z</updated>
    <id>https://example.com/blog/resize-kali-linux-root-filesystem/</id>
    <content type="html">&lt;p&gt;I recently ran out of space (after installing one too many tools) on a Kali Linux VM I&#39;m using for learning and experimentation.&lt;/p&gt;&lt;p&gt;The machine is running in KVM via libvirt on a Linux host, but instructions will be similar when using different virtualization models.&lt;/p&gt;&lt;p&gt;If you&#39;re already familiar with this and just want the commands, &lt;a href=&quot;https://example.com/blog/resize-kali-linux-root-filesystem/#tl-dr-version&quot;&gt;jump to the TL;DR version&lt;/a&gt; below.&lt;/p&gt;&lt;h1 id=&quot;resize-the-virtual-disk-image&quot;&gt;Resize the virtual disk image&lt;/h1&gt;&lt;p&gt;Make sure the guest VM is stopped:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;virsh&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;shutdown&lt;/span&gt; kali
&lt;span class=&quot;token comment&quot;&gt;# Or `sudo shutdown -h now` from the guest&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Head to the location where the libvirt disk image is stored:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token builtin class-name&quot;&gt;cd&lt;/span&gt; /var/lib/libvirt/images&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;(optional) Double check the current disk size:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;% &lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; qemu-img info kali.qcow2
&lt;span class=&quot;token comment&quot;&gt;# [...]&lt;/span&gt;
virtual size: &lt;span class=&quot;token number&quot;&gt;20&lt;/span&gt; GiB &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;21474836480&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Resize the disk image:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; qemu-img resize kali.qcow2  40G  &lt;span class=&quot;token comment&quot;&gt;# absolute&lt;/span&gt;
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; qemu-img resize kali.qcow2 +20G  &lt;span class=&quot;token comment&quot;&gt;# relative, add 20G&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h1 id=&quot;log-into-the-vm-and-resize-the-root-partition&quot;&gt;Log into the VM and resize the root partition&lt;/h1&gt;&lt;p&gt;Start the libvirt guest:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;virsh&lt;/span&gt; start kali&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Once the machine has booted up, connect to it and list the partitions on the root drive.&lt;/p&gt;&lt;p&gt;If you selected the default partitioning type (no LVM or encryption, all files in one partition) during installation, it should look something like this:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ &lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;fdisk&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-l&lt;/span&gt; /dev/vda
Disk /dev/vda: &lt;span class=&quot;token number&quot;&gt;40&lt;/span&gt; GiB, &lt;span class=&quot;token number&quot;&gt;42949672960&lt;/span&gt; bytes, &lt;span class=&quot;token number&quot;&gt;83886080&lt;/span&gt; sectors
Units: sectors of &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; * &lt;span class=&quot;token number&quot;&gt;512&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;512&lt;/span&gt; bytes
Sector size &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;logical/physical&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;: &lt;span class=&quot;token number&quot;&gt;512&lt;/span&gt; bytes / &lt;span class=&quot;token number&quot;&gt;512&lt;/span&gt; bytes
I/O size &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;minimum/optimal&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;: &lt;span class=&quot;token number&quot;&gt;512&lt;/span&gt; bytes / &lt;span class=&quot;token number&quot;&gt;512&lt;/span&gt; bytes
Disklabel type: dos
Disk identifier: 0xa9ac9489

Device     Boot    Start      End  Sectors  Size Id Type
/dev/vda1  *        &lt;span class=&quot;token number&quot;&gt;2048&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;39684095&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;39682048&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;18&lt;/span&gt;.9G &lt;span class=&quot;token number&quot;&gt;83&lt;/span&gt; Linux
/dev/vda2       &lt;span class=&quot;token number&quot;&gt;39686142&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;41940991&lt;/span&gt;  &lt;span class=&quot;token number&quot;&gt;2254850&lt;/span&gt;  &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt;.1G  f W95 Ext&#39;d &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;LBA&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
/dev/vda5       &lt;span class=&quot;token number&quot;&gt;39686144&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;41940991&lt;/span&gt;  &lt;span class=&quot;token number&quot;&gt;2254848&lt;/span&gt;  &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt;.1G &lt;span class=&quot;token number&quot;&gt;82&lt;/span&gt; Linux swap / Solaris&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&lt;li&gt;Partition 1 is the main partition containing the Linux root filesystem&lt;/li&gt;&lt;li&gt;Partition 2 is an extended partition, containing logical partitions&lt;/li&gt;&lt;li&gt;Partition 5 is a swap (logical) partition&lt;/li&gt;&lt;/ul&gt;&lt;div class=&quot;callout callout-danger&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;[!] Danger&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;If your partitions layout doesn&#39;t look like the example above, (for example you have an extra partition for &lt;code&gt;/home&lt;/code&gt;, or are using LVM and/or LUKS), proceeding with these instructions will likely result in data loss, and a non-functioning machine.&lt;/p&gt;&lt;ul&gt;&lt;li&gt;&lt;p&gt;If you used LVM, you should be able to extend the root logical volume using &lt;code&gt;lvchange&lt;/code&gt;. Space will be allocated from the available space at the end of the drive, and extra logical volumes won&#39;t be affected.&lt;/p&gt;&lt;/li&gt;&lt;li&gt;&lt;p&gt;If you have extra (physical or logical) partitions and no LVM, your best option is to back them up, destroy, and recreate them. We won&#39;t be covering how to do that in this guide though.&lt;/p&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/div&gt;&lt;/div&gt;&lt;p&gt;Disable swap, so we can remove the partition:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;swapoff /dev/vda5
&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;(we will have to recreate it later, but thankfully it doesn&#39;t contain permanent data).&lt;/p&gt;&lt;p&gt;Use &lt;code&gt;fdisk&lt;/code&gt; to modify the partition table.&lt;/p&gt;&lt;p&gt;What we want to do is:&lt;/p&gt;&lt;ul&gt;&lt;li&gt;Delete all partitions from the drive&lt;/li&gt;&lt;li&gt;Recreate a new root partition with sufficient space, &lt;strong&gt;MAKING SURE IT STARTS AT THE EXACT SAME BLOCK AS THE PREVIOUS ONE&lt;/strong&gt;, so the filesystem is preserved as well.&lt;/li&gt;&lt;li&gt;Recreate the swap partition&lt;/li&gt;&lt;/ul&gt;&lt;h2 id=&quot;start-fdisk&quot;&gt;Start fdisk&lt;/h2&gt;&lt;p&gt;Start &lt;code&gt;fdisk&lt;/code&gt;:&lt;/p&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;$ sudo fdisk /dev/vda

Welcome to fdisk (util-linux 2.41.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

This disk is currently in use - repartitioning is probably a bad idea.
It&#39;s recommended to umount all file systems, and swapoff all swap
partitions on this disk.&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Ignore the warning, we like to live dangerously.&lt;/p&gt;&lt;h2 id=&quot;verify-partition-table&quot;&gt;Verify partition table&lt;/h2&gt;&lt;p&gt;It should match what we got from &lt;code&gt;fdisk -l&lt;/code&gt; earlier.&lt;/p&gt;&lt;p&gt;Make sure to note down the &lt;code&gt;Start&lt;/code&gt; sector for the first (&lt;code&gt;/dev/vda1&lt;/code&gt;) partition, as we&#39;ll need to double-check it later.&lt;/p&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): p

Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa9ac9489

Device     Boot    Start      End  Sectors  Size Id Type
/dev/vda1  *        2048 39684095 39682048 18.9G 83 Linux
/dev/vda2       39686142 41940991  2254850  1.1G  f W95 Ext&#39;d (LBA)
/dev/vda5       39686144 41940991  2254848  1.1G 82 Linux swap / Solaris&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;verify-free-space&quot;&gt;Verify free space&lt;/h2&gt;&lt;p&gt;Make sure we have enough space at the end of the drive:&lt;/p&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): F

Unpartitioned space /dev/vda: 20 GiB, 21475885056 bytes, 41945088 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

   Start      End  Sectors Size
41940992 83886079 41945088  20G&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;delete-old-partitions&quot;&gt;Delete old partitions&lt;/h2&gt;&lt;div class=&quot;callout callout-warning&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;&amp;lt;!&amp;gt; Warning&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;While &lt;code&gt;fdisk&lt;/code&gt; offers a command to create a new partition table, doing so will also remove the bootloader from the main disk, requiring extra steps to reinstall. The quickest way is to just use the &lt;code&gt;d&lt;/code&gt; command to delete all partitions one by one instead.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): d
Partition number (1,2,5, default 5): 5

Partition 5 has been deleted.

Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;create-a-new-root-partition&quot;&gt;Create a new root partition&lt;/h2&gt;&lt;div class=&quot;callout callout-warning&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;&amp;lt;!&amp;gt; Important&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;Make sure the &lt;code&gt;First sector&lt;/code&gt; for the new partition is the same as the old partition, so the filesystem will be preserved.&lt;/p&gt;&lt;p&gt;Also remember to leave some space for the swap partition at the end (2 GB in this case, adjust as needed).&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-83886079, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-83886079, default 83886079): -2G

Created a new partition 1 of type &#39;Linux&#39; and of size 38 GiB.
Partition #1 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: n&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;create-a-new-swap-partition&quot;&gt;Create a new swap partition&lt;/h2&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): e
Partition number (2-4, default 2):
First sector (79691776-83886079, default 79691776):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (79691776-83886079, default 83886079):

Created a new partition 2 of type &#39;Extended&#39; and of size 2 GiB.

Command (m for help): n
All space for primary partitions is in use.
Adding logical partition 5
First sector (79693824-83886079, default 79693824):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (79693824-83886079, default 83886079):

Created a new partition 5 of type &#39;Linux&#39; and of size 2 GiB.

Command (m for help): t
Partition number (1,2,5, default 5): 5
Hex code or alias (type L to list all): swap

Changed type of partition &#39;Linux&#39; to &#39;Linux swap / Solaris&#39;.&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;double-check-the-new-partitioning-layout&quot;&gt;Double-check the new partitioning layout&lt;/h2&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): p
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xae55699d

Device     Boot    Start      End  Sectors Size Id Type
/dev/vda1           2048 79691775 79689728  38G 83 Linux
/dev/vda2       79691776 83886079  4194304   2G  5 Extended
/dev/vda5       79693824 83886079  4192256   2G 82 Linux swap / Solaris&lt;/code&gt;&lt;/pre&gt;&lt;div class=&quot;callout callout-note&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;[*] Note&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;If you made any mistakes, you can quit with &lt;code&gt;q&lt;/code&gt; and start over.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;h2 id=&quot;write-changes-to-disk&quot;&gt;Write changes to disk&lt;/h2&gt;&lt;pre class=&quot;language-text&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;Command (m for help): w
The partition table has been altered.
Failed to remove partition 5 from system: Device or resource busy
Failed to update system information about partition 1: Device or resource busy
Failed to add partition 5 to system: Device or resource busy

The kernel still uses the old partitions. The new table will be used at the next reboot.
Syncing disks.&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;reboot&quot;&gt;Reboot&lt;/h2&gt;&lt;p&gt;Reboot is needed to reload the partition table.&lt;/p&gt;&lt;h1 id=&quot;resize-the-ext4-filesystem&quot;&gt;Resize the ext4 filesystem&lt;/h1&gt;&lt;p&gt;This will make it use all the available space on the partition:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; resize2fs /dev/vda1&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If you run &lt;code&gt;df -h /&lt;/code&gt;, you should now see the increased filesystem capacity.&lt;/p&gt;&lt;h1 id=&quot;recreate-swap-partition&quot;&gt;Recreate swap partition&lt;/h1&gt;&lt;p&gt;For convenience, reuse the same UUID as the previous swap partition we deleted earlier (you can get it from &lt;code&gt;/etc/fstab&lt;/code&gt;):&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;mkswap&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-U&lt;/span&gt; 2712e819-33fa-4d79-af30-74f91bf93ca8 /dev/vda5
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;swapon&lt;/span&gt; /dev/vda5&lt;/code&gt;&lt;/pre&gt;&lt;h1 id=&quot;tl-dr-version&quot;&gt;TL;DR version&lt;/h1&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token comment&quot;&gt;# Host: shutdown VM&lt;/span&gt;
&lt;span class=&quot;token function&quot;&gt;virsh&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;shutdown&lt;/span&gt; kali

&lt;span class=&quot;token comment&quot;&gt;# Host: resize disk&lt;/span&gt;
qemu-img resize kali.qcow2 +20G

&lt;span class=&quot;token comment&quot;&gt;# Host: restart VM&lt;/span&gt;
&lt;span class=&quot;token function&quot;&gt;virsh&lt;/span&gt; start kali

&lt;span class=&quot;token comment&quot;&gt;# Guest: recreate partitions&lt;/span&gt;
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;fdisk&lt;/span&gt; /dev/vda
&lt;span class=&quot;token comment&quot;&gt;# - Delete all partitions using the `d` command.&lt;/span&gt;
&lt;span class=&quot;token comment&quot;&gt;#   DO NOT create a new partition table using the `o` command.&lt;/span&gt;
&lt;span class=&quot;token comment&quot;&gt;# - Create a new primary partition:&lt;/span&gt;
n  &lt;span class=&quot;token comment&quot;&gt;# new partition&lt;/span&gt;
p  &lt;span class=&quot;token comment&quot;&gt;# primary&lt;/span&gt;
&lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt;  &lt;span class=&quot;token comment&quot;&gt;# first partition&lt;/span&gt;
First sector: &lt;span class=&quot;token number&quot;&gt;2048&lt;/span&gt;  &lt;span class=&quot;token comment&quot;&gt;# Must match what copied from the previous partition!&lt;/span&gt;
Last sector: &lt;span class=&quot;token parameter variable&quot;&gt;-2G&lt;/span&gt;    &lt;span class=&quot;token comment&quot;&gt;# Leave 2 GB at the end&lt;/span&gt;

Created a new partition &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; of &lt;span class=&quot;token builtin class-name&quot;&gt;type&lt;/span&gt; &lt;span class=&quot;token string&quot;&gt;&#39;Linux&#39;&lt;/span&gt; and of size &lt;span class=&quot;token number&quot;&gt;38&lt;/span&gt; GiB.
Partition &lt;span class=&quot;token comment&quot;&gt;#1 contains a ext4 signature.&lt;/span&gt;

Do you want to remove the signature? &lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Y&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;es/&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;N&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;o: n  &lt;span class=&quot;token comment&quot;&gt;# &amp;lt;-- IMPORTANT!&lt;/span&gt;

&lt;span class=&quot;token comment&quot;&gt;# - Create a new swap partition&lt;/span&gt;
n  &lt;span class=&quot;token comment&quot;&gt;# new partition&lt;/span&gt;
e  &lt;span class=&quot;token comment&quot;&gt;# extended&lt;/span&gt;
&lt;span class=&quot;token comment&quot;&gt;# (accept defaults for the rest)&lt;/span&gt;

n  &lt;span class=&quot;token comment&quot;&gt;# new partition&lt;/span&gt;
&lt;span class=&quot;token comment&quot;&gt;# (accept defaults for the rest)&lt;/span&gt;

t  &lt;span class=&quot;token comment&quot;&gt;# set partition type&lt;/span&gt;
Partition number: &lt;span class=&quot;token number&quot;&gt;5&lt;/span&gt;
Hex code or alias: swap

p  &lt;span class=&quot;token comment&quot;&gt;# double-check&lt;/span&gt;
w  &lt;span class=&quot;token comment&quot;&gt;# write&lt;/span&gt;

&lt;span class=&quot;token comment&quot;&gt;# Guest: restart to reload partition table&lt;/span&gt;
&lt;span class=&quot;token function&quot;&gt;reboot&lt;/span&gt;

&lt;span class=&quot;token comment&quot;&gt;# Guest: resize filesystem&lt;/span&gt;
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; resize2fs /dev/vda1
&lt;span class=&quot;token function&quot;&gt;df&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-h&lt;/span&gt; /  &lt;span class=&quot;token comment&quot;&gt;# check updated size&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;</content>
  </entry>
  <entry>
    <title>Compiling Emacs 30 on Ubuntu 24.04</title>
    <link href="https://example.com/blog/compile-emacs-on-ubuntu/" />
    <updated>2026-01-28T22:19:19Z</updated>
    <id>https://example.com/blog/compile-emacs-on-ubuntu/</id>
    <content type="html">&lt;h1 id=&quot;install-dependencies&quot;&gt;Install dependencies&lt;/h1&gt;&lt;p&gt;First of all, make sure to enable &lt;code&gt;deb-src&lt;/code&gt; in &lt;code&gt;/etc/apt/sources.list.d/ubuntu.sources&lt;/code&gt; (&lt;code&gt;Types: deb deb-src&lt;/code&gt;), then run &lt;code&gt;apt update&lt;/code&gt;.&lt;/p&gt;&lt;p&gt;Install build dependencies:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;apt&lt;/span&gt; build-dep &lt;span class=&quot;token parameter variable&quot;&gt;-y&lt;/span&gt; emacs
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;apt&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;install&lt;/span&gt; libtree-sitter-dev&lt;/code&gt;&lt;/pre&gt;&lt;h1 id=&quot;clone-the-source-repo&quot;&gt;Clone the source repo&lt;/h1&gt;&lt;p&gt;If you only want version 30:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;git&lt;/span&gt; clone &lt;span class=&quot;token parameter variable&quot;&gt;--depth&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;--branch&lt;/span&gt; emacs-30 git://git.savannah.gnu.org/emacs.git
&lt;span class=&quot;token builtin class-name&quot;&gt;cd&lt;/span&gt; ./emacs&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If you want the entire emacs repo (beware, it&#39;s fairly large!):&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;git&lt;/span&gt; clone git://git.savannah.gnu.org/emacs.git
&lt;span class=&quot;token builtin class-name&quot;&gt;cd&lt;/span&gt; ./emacs
&lt;span class=&quot;token function&quot;&gt;git&lt;/span&gt; worktree &lt;span class=&quot;token function&quot;&gt;add&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;/emacs-30 origin/emacs-30&lt;/code&gt;&lt;/pre&gt;&lt;h1 id=&quot;configure&quot;&gt;Configure&lt;/h1&gt;&lt;p&gt;Run &lt;code&gt;autogen&lt;/code&gt;:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;./autogen.sh&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Run &lt;code&gt;./configure&lt;/code&gt;:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token assign-left variable&quot;&gt;CFLAGS&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token string&quot;&gt;&quot;-march=native -pipe&quot;&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;&#92;&lt;/span&gt;
    ./configure &lt;span class=&quot;token punctuation&quot;&gt;&#92;&lt;/span&gt;
    &lt;span class=&quot;token parameter variable&quot;&gt;--prefix&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;/opt/emacs30 &lt;span class=&quot;token punctuation&quot;&gt;&#92;&lt;/span&gt;
    --with-cairo --with-libsystemd --with-modules &lt;span class=&quot;token punctuation&quot;&gt;&#92;&lt;/span&gt;
    --with-tree-sitter &lt;span class=&quot;token punctuation&quot;&gt;&#92;&lt;/span&gt;
    --with-x-toolkit&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;gtk3&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&lt;li&gt;Set &lt;code&gt;--prefix&lt;/code&gt; to the desired install location&lt;/li&gt;&lt;li&gt;If you don&#39;t want X support, remove the &lt;code&gt;--with-x-toolkit&lt;/code&gt; option, add &lt;code&gt;--without-x --without-sound&lt;/code&gt;&lt;/li&gt;&lt;li&gt;See &lt;code&gt;./configure --help&lt;/code&gt; for other options&lt;/li&gt;&lt;/ul&gt;&lt;div class=&quot;callout callout-warning&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;&amp;lt;!&amp;gt; Warning&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;The &lt;code&gt;-march=native&lt;/code&gt; flag will tell gcc to make the resulting executable compatible &lt;em&gt;only&lt;/em&gt; with the architecture of the current CPU. Remove (or modify) it if you need to redistribute the executable.&lt;/p&gt;&lt;p&gt;See also &lt;a href=&quot;https://wiki.gentoo.org/wiki/GCC_optimization#-march&quot;&gt;&lt;code&gt;-march&lt;/code&gt; on the Gentoo wiki&lt;/a&gt;.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;h1 id=&quot;build&quot;&gt;Build&lt;/h1&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;make&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-j24&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&lt;li&gt;Adjust the &lt;code&gt;-j&lt;/code&gt; option based on the number of cores on your machine&lt;/li&gt;&lt;/ul&gt;&lt;h1 id=&quot;install&quot;&gt;Install&lt;/h1&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;make&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;install&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h1 id=&quot;run&quot;&gt;Run&lt;/h1&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;/opt/emacs30/bin/emacs
/opt/emacs30/bin/emacsclient&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Add to path:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token assign-left variable&quot;&gt;&lt;span class=&quot;token environment constant&quot;&gt;PATH&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token string&quot;&gt;&quot;/opt/emacs30/bin:&lt;span class=&quot;token variable&quot;&gt;${&lt;span class=&quot;token environment constant&quot;&gt;PATH&lt;/span&gt;}&lt;/span&gt;&quot;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h1 id=&quot;see-also&quot;&gt;See also&lt;/h1&gt;&lt;ul&gt;&lt;li&gt;&lt;a href=&quot;https://gitlab.archlinux.org/archlinux/packaging/packages/emacs/-/blob/main/PKGBUILD&quot;&gt;Archlinux PKGBUILD for Emacs&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href=&quot;https://wiki.gentoo.org/wiki/GCC_optimization&quot;&gt;GCC Optimization&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;</content>
  </entry>
  <entry>
    <title>Linux: automount LUKS device</title>
    <link href="https://example.com/blog/linux-automount-luks/" />
    <updated>2026-01-17T12:52:16Z</updated>
    <id>https://example.com/blog/linux-automount-luks/</id>
    <content type="html">&lt;p&gt;This guide was written specifically on Ubuntu, but it should work similarly on most modern Linux distributions.&lt;/p&gt;&lt;p&gt;Let&#39;s assume we have an existing device, formatted using LUKS and containing an ext4 filesystem. We want to automount it at boot, at a specific mount point location.&lt;/p&gt;&lt;h2 id=&quot;find-the-name-and-uuid-of-the-luks-device&quot;&gt;Find the name and uuid of the LUKS device&lt;/h2&gt;&lt;p&gt;List all the LUKS devices in the system:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;% lsblk &lt;span class=&quot;token parameter variable&quot;&gt;--fs&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;--json&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; jq &lt;span class=&quot;token string&quot;&gt;&#39;.blockdevices | .. | select(type == &quot;object&quot; and .fstype == &quot;crypto_LUKS&quot;) | [.name,.fstype,.uuid] | @tsv&#39;&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-r&lt;/span&gt;
sdx crypto_LUKS da2b593e-8c74-445e-aef6-f0b7f656d8c3&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;You can also use the &amp;quot;Disks&amp;quot; application, if you prefer using a GUI.&lt;/p&gt;&lt;h2 id=&quot;enable-key-based-encryption&quot;&gt;Enable key-based encryption&lt;/h2&gt;&lt;p&gt;We need a way to unlock the device without typing in a passphrase. The solution is to use an encryption key stored in a file:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;mkdir&lt;/span&gt; /etc/luks-keys/
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;chmod&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;700&lt;/span&gt; /etc/luks-keys
&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; openssl genrsa &lt;span class=&quot;token parameter variable&quot;&gt;-out&lt;/span&gt; /etc/luks-keys/default &lt;span class=&quot;token number&quot;&gt;2048&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;div class=&quot;callout callout-warning&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;&amp;lt;!&amp;gt; Warning&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;The file can be stored anywhere, but make sure it is only readable by root.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;p&gt;Add the key to the LUKS device:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; cryptsetup luksAddKey /dev/disk/by-uuid/da2b593e-8c74-445e-aef6-f0b7f656d8c3 /etc/luks-keys/default&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;add-an-entry-to-etc-crypttab&quot;&gt;Add an entry to &lt;code&gt;/etc/crypttab&lt;/code&gt;&lt;/h2&gt;&lt;p&gt;This allows the device to be unlocked automatically&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token builtin class-name&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;token string&quot;&gt;&quot;mydata UUID=da2b593e-8c74-445e-aef6-f0b7f656d8c3 /etc/luks-keys/default luks&quot;&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;tee&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-a&lt;/span&gt; /etc/crypttab&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;open-the-newly-added-device&quot;&gt;Open the newly added device&lt;/h2&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; cryptdisks_start mydata&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;create-a-mount-point&quot;&gt;Create a mount point&lt;/h2&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;sudo&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;mkdir&lt;/span&gt; /mnt/mydata&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;add-an-entry-to-etc-fstab&quot;&gt;Add an entry to &lt;code&gt;/etc/fstab&lt;/code&gt;&lt;/h2&gt;&lt;pre&gt;&lt;code&gt;echo /dev/mapper/mydata /mnt/mydata auto defaults 0 0 | sudo tee -a /etc/fstab
&lt;/code&gt;&lt;/pre&gt;&lt;div class=&quot;callout callout-note&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;[*] Note&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This assumes that the LUKS device contains one filesystem.&lt;br&gt;For LVM or more complex partitioning schemes, you might need to tweak things a bit.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;h2 id=&quot;reload-systemd&quot;&gt;Reload systemd&lt;/h2&gt;&lt;p&gt;This will update systemd&#39;s internal fstab definition:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;systemctl daemon-reload&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;mount-the-filesystem&quot;&gt;Mount the filesystem&lt;/h2&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token function&quot;&gt;mount&lt;/span&gt; /mnt/mydata&lt;/code&gt;&lt;/pre&gt;</content>
  </entry>
  <entry>
    <title>Attacking IPSec IKE with insecure PSK</title>
    <link href="https://example.com/blog/attacking-ipsec-ike-insecure-psk/" />
    <updated>2025-12-28T15:25:00Z</updated>
    <id>https://example.com/blog/attacking-ipsec-ike-insecure-psk/</id>
    <content type="html">&lt;div class=&quot;callout callout-note&quot;&gt;&lt;div class=&quot;callout-header&quot;&gt;[*] Note&lt;/div&gt;&lt;div class=&quot;callout-body&quot;&gt;&lt;p&gt;This post describes a scenario I encountered while solving a &amp;quot;box&amp;quot; at &lt;a href=&quot;https://app.hackthebox.com/&quot;&gt;HTB Labs&lt;/a&gt;. As such, it only focuses on the actions taken in the specific case.&lt;/p&gt;&lt;p&gt;For more information about general attacks to IPSec and IKE, see the reference links at the end of the page.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;h2 id=&quot;discovery&quot;&gt;Discovery&lt;/h2&gt;&lt;p&gt;A port scan reveals port &lt;code&gt;500/udp&lt;/code&gt; is open:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ nmap &lt;span class=&quot;token parameter variable&quot;&gt;-sU&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$TARGET&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-T5&lt;/span&gt;
PORT      STATE  SERVICE
&lt;span class=&quot;token number&quot;&gt;500&lt;/span&gt;/udp   &lt;span class=&quot;token function&quot;&gt;open&lt;/span&gt;   isakmp&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The port is commonly used by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Internet_Security_Association_and_Key_Management_Protocol&quot;&gt;ISAKMP&lt;/a&gt; service, part of the &lt;a href=&quot;https://en.wikipedia.org/wiki/IPsec&quot;&gt;IPSec&lt;/a&gt; suite and responsible for handling key exchange through the &lt;a href=&quot;https://en.wikipedia.org/wiki/Internet_Key_Exchange&quot;&gt;IKE&lt;/a&gt; service.&lt;/p&gt;&lt;h2 id=&quot;probing&quot;&gt;Probing&lt;/h2&gt;&lt;p&gt;Using &lt;a href=&quot;https://github.com/royhills/ike-scan&quot;&gt;ike-scan&lt;/a&gt;, we can get more information about the endpoint, and the supported transformations:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ ike-scan &lt;span class=&quot;token parameter variable&quot;&gt;-M&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$TARGET&lt;/span&gt;
Starting ike-scan &lt;span class=&quot;token number&quot;&gt;1.9&lt;/span&gt;.6 with &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; hosts &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;http://www.nta-monitor.com/tools/ike-scan/&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;token number&quot;&gt;10.10&lt;/span&gt;.123.436	Main Mode Handshake returned
	&lt;span class=&quot;token assign-left variable&quot;&gt;HDR&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;CKY-R&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;a5b21e95b0e85fd9&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;SA&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Enc&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;3DES &lt;span class=&quot;token assign-left variable&quot;&gt;Hash&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;SHA1 &lt;span class=&quot;token assign-left variable&quot;&gt;Group&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;2&lt;/span&gt;:modp1024 &lt;span class=&quot;token assign-left variable&quot;&gt;Auth&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;PSK &lt;span class=&quot;token assign-left variable&quot;&gt;LifeType&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;Seconds &lt;span class=&quot;token assign-left variable&quot;&gt;LifeDuration&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;28800&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;VID&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;09002689dfd6b712 &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;XAUTH&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;VID&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;afcad71368a1f1c96b8696fc77570100 &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Dead Peer Detection v1.0&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

Ending ike-scan &lt;span class=&quot;token number&quot;&gt;1.9&lt;/span&gt;.6: &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; hosts scanned &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0.044&lt;/span&gt; seconds &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;22.53&lt;/span&gt; hosts/sec&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;.  &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; returned handshake&lt;span class=&quot;token punctuation&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt; returned notify&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Also check &amp;quot;aggressive mode&amp;quot;, which, when using PSK authentication, will return a plain text hash we can use for offline cracking:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ ike-scan &lt;span class=&quot;token parameter variable&quot;&gt;-M&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-A&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-P&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$TARGET&lt;/span&gt;
Starting ike-scan &lt;span class=&quot;token number&quot;&gt;1.9&lt;/span&gt;.6 with &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; hosts &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;http://www.nta-monitor.com/tools/ike-scan/&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;token number&quot;&gt;10.10&lt;/span&gt;.123.436	Aggressive Mode Handshake returned
	&lt;span class=&quot;token assign-left variable&quot;&gt;HDR&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;CKY-R&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;02b63ef4bd664662&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;SA&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Enc&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;3DES &lt;span class=&quot;token assign-left variable&quot;&gt;Hash&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;SHA1 &lt;span class=&quot;token assign-left variable&quot;&gt;Group&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;2&lt;/span&gt;:modp1024 &lt;span class=&quot;token assign-left variable&quot;&gt;Auth&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;PSK &lt;span class=&quot;token assign-left variable&quot;&gt;LifeType&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;Seconds &lt;span class=&quot;token assign-left variable&quot;&gt;LifeDuration&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;28800&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	KeyExchange&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;128&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	Nonce&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;32&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	ID&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Type&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;ID_USER_FQDN, &lt;span class=&quot;token assign-left variable&quot;&gt;Value&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;someuser@example.com&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;VID&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;09002689dfd6b712 &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;XAUTH&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;VID&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;afcad71368a1f1c96b8696fc77570100 &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Dead Peer Detection v1.0&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	Hash&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;20&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Normally, we would need to use various bruteforcing techniques to determine a valid client ID to use for the key exchange process.&lt;/p&gt;&lt;p&gt;In this scenario, it appears any client ID is accepted by the server.&lt;/p&gt;&lt;p&gt;Let&#39;s retrieve our connection parameters (including the hash) and store them to a file:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ ike-scan &lt;span class=&quot;token parameter variable&quot;&gt;-M&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-A&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-Pparams.txt&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$TARGET&lt;/span&gt;
Starting ike-scan &lt;span class=&quot;token number&quot;&gt;1.9&lt;/span&gt;.6 with &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; hosts &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;http://www.nta-monitor.com/tools/ike-scan/&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;token number&quot;&gt;10.10&lt;/span&gt;.123.456	Aggressive Mode Handshake returned
	&lt;span class=&quot;token assign-left variable&quot;&gt;HDR&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;CKY-R&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;e7c8381dccab3ea1&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;SA&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Enc&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;3DES &lt;span class=&quot;token assign-left variable&quot;&gt;Hash&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;SHA1 &lt;span class=&quot;token assign-left variable&quot;&gt;Group&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;2&lt;/span&gt;:modp1024 &lt;span class=&quot;token assign-left variable&quot;&gt;Auth&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;PSK &lt;span class=&quot;token assign-left variable&quot;&gt;LifeType&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;Seconds &lt;span class=&quot;token assign-left variable&quot;&gt;LifeDuration&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;28800&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	KeyExchange&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;128&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	Nonce&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;32&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	ID&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Type&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;ID_USER_FQDN, &lt;span class=&quot;token assign-left variable&quot;&gt;Value&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;user@example.com&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;VID&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;09002689dfd6b712 &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;XAUTH&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;token assign-left variable&quot;&gt;VID&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;afcad71368a1f1c96b8696fc77570100 &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;Dead Peer Detection v1.0&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
	Hash&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;20&lt;/span&gt; bytes&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

Ending ike-scan &lt;span class=&quot;token number&quot;&gt;1.9&lt;/span&gt;.6: &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; hosts scanned &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0.060&lt;/span&gt; seconds &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;16.69&lt;/span&gt; hosts/sec&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;.  &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; returned handshake&lt;span class=&quot;token punctuation&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt; returned notify&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;We can then use &lt;code&gt;psk-crack&lt;/code&gt; (from the &lt;a href=&quot;https://github.com/royhills/ike-scan&quot;&gt;ike-scan&lt;/a&gt; suite) to crack the hash:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ psk-crack params.txt &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;gunzip &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; /usr/share/wordlists/rockyou.txt.gz&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
Starting psk-crack &lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;ike-scan &lt;span class=&quot;token number&quot;&gt;1.9&lt;/span&gt;.6&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;http://www.nta-monitor.com/tools/ike-scan/&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
Running &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; dictionary cracking mode
key &lt;span class=&quot;token string&quot;&gt;&quot;secretpassword&quot;&lt;/span&gt; matches SHA1 &lt;span class=&quot;token builtin class-name&quot;&gt;hash&lt;/span&gt; 6adfb183a4a2c94a2f92dab5ade762a47889a5a1
Ending psk-crack: &lt;span class=&quot;token number&quot;&gt;8045040&lt;/span&gt; iterations &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;3.159&lt;/span&gt; seconds &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;2546545.85&lt;/span&gt; iterations/sec&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Or hashcat (use &lt;code&gt;hashcat -hh&lt;/code&gt; to list hash types, search for IKE-SHA1):&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;$ hashcat &lt;span class=&quot;token parameter variable&quot;&gt;-m&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;5400&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-a&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt; params.txt /usr/share/wordlists/rockyou.txt.gz
&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.:6adfb183a4a2c94a2f92dab5ade762a47889a5a1:secretpassword
&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;We could now use &lt;code&gt;vpnc&lt;/code&gt; to attempt to connect to the IPSec vpn.&lt;/p&gt;&lt;p&gt;Or perhaps attempt to reuse the gathered credentials to access some different service on the same system.&lt;/p&gt;&lt;h2 id=&quot;reference-links&quot;&gt;Reference links&lt;/h2&gt;&lt;ul&gt;&lt;li&gt;&lt;a href=&quot;https://book.hacktricks.wiki/en/network-services-pentesting/ipsec-ike-vpn-pentesting.html&quot;&gt;HackTricks: Pentesting IPsec/IKE VPN&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;</content>
  </entry>
  <entry>
    <title>Trick: combining wordlists using standard Unix tools</title>
    <link href="https://example.com/blog/combining-wordlists/" />
    <updated>2025-12-28T13:42:26Z</updated>
    <id>https://example.com/blog/combining-wordlists/</id>
    <content type="html">&lt;p&gt;When performing a dictionary-based brute force attack, as part of a pentesting campaign, it often arises the need to combine multiple wordlists from various sources.&lt;/p&gt;&lt;p&gt;(A common scenario: we want to combine generic wordlists with target-specific words we discovered during information gathering).&lt;/p&gt;&lt;p&gt;We would like to be able to quickly combine them, removing duplicates, and possibly filter them based on some condition (eg. we know the password starts with &amp;quot;&lt;code&gt;foo&lt;/code&gt;&amp;quot;, or the last four digits of a PIN are &lt;code&gt;1234&lt;/code&gt;).&lt;/p&gt;&lt;p&gt;Some wordlists might be fairly large (&lt;code&gt;rockyou.txt.gz&lt;/code&gt; is a good example), so they&#39;re generally best kept around compressed using gzip or other formats.&lt;/p&gt;&lt;p&gt;While some tools support reading gzipped wordlists, this is not true for every tool out there, and certainly this becomes a problem when we want to combine multiple gzipped and non-gzipped wordlists.&lt;/p&gt;&lt;p&gt;Thankfully, using standard Unix shell features, we can quickly generate wordlists on the fly, as needed.&lt;/p&gt;&lt;h2 id=&quot;passing-gzipped-wordlists-to-non-gzip-aware-tools&quot;&gt;Passing gzipped wordlists to non-gzip-aware tools&lt;/h2&gt;&lt;p&gt;Some commands are able to detect compressed files and automatically decompress them (&lt;code&gt;hashcat&lt;/code&gt; for example), but others just expect a plain text file.&lt;/p&gt;&lt;p&gt;Bash (and Zsh) shells support passing the output of a command pipeline to another command as the path to an open file descriptor.&lt;/p&gt;&lt;p&gt;So for example, if we want to use a gzipped wordlist with &lt;code&gt;psk-crack&lt;/code&gt;, we can call it like this:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;psk-crack &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;gunzip &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; /usr/share/wordlists/rockyou.txt.gz&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Behind the scenes:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;$ &lt;span class=&quot;token function&quot;&gt;ls&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-lh&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token builtin class-name&quot;&gt;echo&lt;/span&gt; hello&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
lr-x------ &lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt; samu samu &lt;span class=&quot;token number&quot;&gt;64&lt;/span&gt; Dec &lt;span class=&quot;token number&quot;&gt;28&lt;/span&gt; 07:48 /proc/self/fd/11 -&lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; &lt;span class=&quot;token string&quot;&gt;&#39;pipe:[1093239]&#39;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;combining-wordlists&quot;&gt;Combining wordlists&lt;/h2&gt;&lt;p&gt;Use &lt;code&gt;cat&lt;/code&gt; to combine multiple files, for example:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;&lt;span class=&quot;token function&quot;&gt;cat&lt;/span&gt; words.txt discovered.txt &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;gzip&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; huge.txt.gz&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;xz &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; huge2.txt.xz&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;removing-duplicate-entries&quot;&gt;Removing duplicate entries&lt;/h2&gt;&lt;p&gt;Use &lt;code&gt;sort -u&lt;/code&gt;, for example:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;sometool &lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;. &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;cat&lt;/span&gt; words.txt &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;gzip&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; words2.txt&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;sort&lt;/span&gt; -u&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;filtering-entries&quot;&gt;Filtering entries&lt;/h2&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;sometool &lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;. &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;cat&lt;/span&gt; words.txt &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;gzip&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; words2.txt&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;grep&lt;/span&gt; &lt;span class=&quot;token string&quot;&gt;&#39;^foobar&#39;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;word-of-caution-not-all-programs-can-handle-pipes&quot;&gt;Word of caution: not all programs can handle pipes&lt;/h2&gt;&lt;p&gt;Some programs such as &lt;code&gt;hashcat&lt;/code&gt; don&#39;t seem to be working properly when reading a wordlist from a pipe:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;$ hashcat &lt;span class=&quot;token parameter variable&quot;&gt;-m&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;5400&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-a&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt; params.txt &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;gunzip &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; /usr/share/wordlists/rockyou.txt.gz&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
hashcat &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;v7.1.2&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; starting
&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;

The wordlist or mask that you are using is too small.
This means that hashcat cannot use the full parallel power of your device&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;s&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;.
Hashcat is expecting at least &lt;span class=&quot;token number&quot;&gt;4096&lt;/span&gt; base words but only got &lt;span class=&quot;token number&quot;&gt;0.0&lt;/span&gt;% of that.
Unless you supply &lt;span class=&quot;token function&quot;&gt;more&lt;/span&gt; work, your cracking speed will drop.
For tips on supplying &lt;span class=&quot;token function&quot;&gt;more&lt;/span&gt; work, see: https://hashcat.net/faq/morework
&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If this is the case, you might have to write the generated wordlist to a temporary file first:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;&lt;span class=&quot;token function&quot;&gt;cat&lt;/span&gt; words.txt &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;gzip&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; words2.txt&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;sort&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-u&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; /tmp/words.txt
hashcat &lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;. /tmp/words.txt&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Or, in the case of &lt;code&gt;hashcat&lt;/code&gt;, you can pipe the wordlist directly to the program input (omit the wordlist parameter in this case):&lt;/p&gt;&lt;pre class=&quot;language-bash&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;&lt;span class=&quot;token function&quot;&gt;cat&lt;/span&gt; words.txt &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token function&quot;&gt;gzip&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; words2.txt&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;sort&lt;/span&gt; &lt;span class=&quot;token parameter variable&quot;&gt;-u&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt;  hashcat &lt;span class=&quot;token punctuation&quot;&gt;..&lt;/span&gt;.&lt;/code&gt;&lt;/pre&gt;</content>
  </entry>
  <entry>
    <title>Parallelize work in Python with Asyncio</title>
    <link href="https://example.com/blog/parallelize-work-python-asyncio/" />
    <updated>2025-12-20T21:30:00Z</updated>
    <id>https://example.com/blog/parallelize-work-python-asyncio/</id>
    <content type="html">&lt;p&gt;Say we have a large amount of I/O-bound tasks (eg. network requests) which we want to run.&lt;/p&gt;&lt;p&gt;We don&#39;t really care about what order we get results back in, but we want to make requests in parallel, in order to speed things up.&lt;/p&gt;&lt;p&gt;A good candidate for doing this in Python is asynchronous code, using coroutines and the &lt;a href=&quot;https://docs.python.org/3/library/asyncio.html&quot;&gt;asyncio&lt;/a&gt; module.&lt;/p&gt;&lt;h2 id=&quot;desired-interface&quot;&gt;Desired interface&lt;/h2&gt;&lt;p&gt;We would like to have a resusable function taking an action and an iterable, and returning an iterable with the results:&lt;/p&gt;&lt;pre class=&quot;language-python&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; parallelize&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;T&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;
    action&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; Callable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; Coroutine&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token boolean&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token boolean&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; T&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    argslist&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; Iterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Iterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Any&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; AsyncIterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Iterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Any&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    numworkers&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;token builtin&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; AsyncIterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;T&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;token keyword&quot;&gt;pass&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Usage example:&lt;/p&gt;&lt;pre class=&quot;language-python&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;
&lt;span class=&quot;token comment&quot;&gt;# List of URLs we want to retrieve&lt;/span&gt;
urls &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;

&lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;with&lt;/span&gt; aiohttp&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;ClientSession&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;as&lt;/span&gt; session&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;

    &lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;action&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;url&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;with&lt;/span&gt; session&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;get&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;url&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;as&lt;/span&gt; resp&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
            &lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; resp&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;read&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
            &lt;span class=&quot;token keyword&quot;&gt;return&lt;/span&gt; resp

    &lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;for&lt;/span&gt; resp &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; parallelize&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;action&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; urls&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; numworkers&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;100&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token string-interpolation&quot;&gt;&lt;span class=&quot;token string&quot;&gt;f&quot;&lt;/span&gt;&lt;span class=&quot;token interpolation&quot;&gt;&lt;span class=&quot;token punctuation&quot;&gt;{&lt;/span&gt;resp&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;status&lt;span class=&quot;token punctuation&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;token string&quot;&gt; &lt;/span&gt;&lt;span class=&quot;token interpolation&quot;&gt;&lt;span class=&quot;token punctuation&quot;&gt;{&lt;/span&gt;resp&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;url&lt;span class=&quot;token punctuation&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;token string&quot;&gt;&quot;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;if&lt;/span&gt; resp&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;ok&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
            &lt;span class=&quot;token keyword&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; resp&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;text&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;implementation&quot;&gt;Implementation&lt;/h2&gt;&lt;p&gt;The main entry point is the &lt;code&gt;parallelize()&lt;/code&gt; function we&#39;ve seen earlier.&lt;/p&gt;&lt;p&gt;It spawns a predefined amount of &amp;quot;workers&amp;quot; as asyncio tasks, connecting them to an input and an output Queue, and providing them with the &lt;code&gt;action&lt;/code&gt; function to call on each set of arguments.&lt;/p&gt;&lt;p&gt;It then watches the queues, continuously yielding results until all the input items have been processed.&lt;/p&gt;&lt;pre class=&quot;language-python&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; parallelize&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;T&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;
    action&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; Callable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; Coroutine&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token boolean&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token boolean&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; T&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    argslist&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; Iterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Iterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Any&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; AsyncIterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Iterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;Any&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    numworkers&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;token builtin&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; AsyncIterable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;T&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;

    inqueue &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Queue&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;maxsize&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;numworkers&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
    outqueue &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Queue&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

    tasks &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;
    &lt;span class=&quot;token keyword&quot;&gt;for&lt;/span&gt; worker_id &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;token builtin&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;numworkers&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        task &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;create_task&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;worker&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;worker_id&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; inqueue&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; outqueue&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; action&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        tasks&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;append&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;task&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

    loader_task &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;create_task&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;queue_loader&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;inqueue&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; argslist&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;token keyword&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token keyword&quot;&gt;not&lt;/span&gt; loader_task&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;done&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;or&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;inqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;_unfinished_tasks &lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;token comment&quot;&gt;# HACK&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;or&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;outqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;_unfinished_tasks &lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;token comment&quot;&gt;# HACK&lt;/span&gt;
    &lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        result &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; outqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;get&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;yield&lt;/span&gt; result
        outqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;task_done&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;token keyword&quot;&gt;for&lt;/span&gt; task &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; tasks&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        task&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;cancel&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;gather&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;*&lt;/span&gt;tasks&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; return_exceptions&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token boolean&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&quot;the-worker-task&quot;&gt;The worker task&lt;/h3&gt;&lt;p&gt;The worker coroutine simply listens on the input queue, executing actions on each set of input arguments, and putting the results back in the output queue:&lt;/p&gt;&lt;pre class=&quot;language-python&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;worker&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;
    worker_id&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;token builtin&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    inqueue&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Queue&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    outqueue&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Queue&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
    action&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; Callable&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; Coroutine&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;token keyword&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;token boolean&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        args &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; inqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;get&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        result &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; action&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;*&lt;/span&gt;args&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        outqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;put_nowait&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;result&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
        inqueue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;task_done&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&quot;loading-the-input-queue&quot;&gt;Loading the input queue&lt;/h3&gt;&lt;p&gt;We could just pre-load the queue before starting the workers, but that could use a significant amount of memory, depending on how input arguments are generated.&lt;/p&gt;&lt;p&gt;For example, the input iterable might actually be a generator, generating millions or billions of URLs to be crawled, so we don&#39;t want to store them in memory all at once.&lt;/p&gt;&lt;p&gt;A more clever way is to use a function to obtain new items and append them to the input queue only as items from the queue get depleted:&lt;/p&gt;&lt;pre class=&quot;language-python&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;queue_loader&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;queue&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; asyncio&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Queue&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; items&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; AsyncIterable &lt;span class=&quot;token operator&quot;&gt;|&lt;/span&gt; Iterable&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;token keyword&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;not&lt;/span&gt; &lt;span class=&quot;token builtin&quot;&gt;isinstance&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;items&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; AsyncIterable&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        items &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; make_aiter&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;items&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;for&lt;/span&gt; item &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; items&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;await&lt;/span&gt; queue&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;put&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;item&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;token comment&quot;&gt;# this blocks when the queue is full&lt;/span&gt;


&lt;span class=&quot;token keyword&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;make_aiter&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;items&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; Iterable&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;token operator&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;&gt;&lt;/span&gt; AsyncIterable&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;token keyword&quot;&gt;for&lt;/span&gt; item &lt;span class=&quot;token keyword&quot;&gt;in&lt;/span&gt; items&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;token keyword&quot;&gt;yield&lt;/span&gt; item&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;make_aiter()&lt;/code&gt; function is just for convenience, so we can indifferently pass either a regular iterable (including a list), or an async iterable.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>Recursive string multidecoder</title>
    <link href="https://example.com/blog/recursive-multidecoder/" />
    <updated>2025-12-20T00:00:00Z</updated>
    <id>https://example.com/blog/recursive-multidecoder/</id>
    <content type="html">&lt;p&gt;It&#39;s common when reverse engineering some piece of software or network protocol, to come across some text that has been encoded multiple times, possibly mixing multiple encodings.&lt;/p&gt;&lt;p&gt;There are several tools that can help with the decoding process, but it can be pretty labor intensive to manually decode a string multiple times, trying to guess the correct encoding in the process.&lt;/p&gt;&lt;p&gt;Wouldn&#39;t it be nice to have a tool to automate this process for us, automatically exploring various combinations of encodings until a result is found?&lt;/p&gt;&lt;h2 id=&quot;tl-dr-just-gimme-the-tool&quot;&gt;TL;DR just gimme the tool&lt;/h2&gt;&lt;p&gt;Sure! Get the source from &lt;a href=&quot;https://github.com/rshk/multidecoder&quot;&gt;github&lt;/a&gt; or &lt;a href=&quot;https://pypi.org/project/text-multidecoder/&quot;&gt;pypi&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Install with &lt;a href=&quot;https://pipx.pypa.io/stable/installation/&quot;&gt;pipx&lt;/a&gt;:&lt;/p&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;&lt;span class=&quot;token comment&quot;&gt;# Latest release&lt;/span&gt;
pipx &lt;span class=&quot;token function&quot;&gt;install&lt;/span&gt; text-multidecoder

&lt;span class=&quot;token comment&quot;&gt;# From a git branch&lt;/span&gt;
pipx &lt;span class=&quot;token function&quot;&gt;install&lt;/span&gt; git+https://github.com/rshk/multidecoder.git@main&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&quot;command-line-usage&quot;&gt;Command line usage&lt;/h3&gt;&lt;pre class=&quot;language-shell&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-shell&quot;&gt;multidecoder &lt;span class=&quot;token parameter variable&quot;&gt;-t&lt;/span&gt; &lt;span class=&quot;token string&quot;&gt;&quot;string to decode&quot;&lt;/span&gt;
multidecoder &lt;span class=&quot;token operator&quot;&gt;&amp;lt;&lt;/span&gt; decodeme.txt&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Example:&lt;/p&gt;&lt;pre class=&quot;language-custom&quot;&gt;
&lt;span class=&quot;token punctuation&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;token keyword&quot;&gt;multidecoder&lt;/span&gt; -t &lt;span class=&quot;token string&quot;&gt;&quot;4a5464434a544979614756736247386c4d6a496c4d30456c4d6a416c4d6a4a5862334a735a4355794d69553352413d3d&quot;&lt;/span&gt;
&lt;span style=&quot;color:#42a5f5&quot;&gt;base64&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;hex[ e1 ae 78 eb 8e 37 e1 ae 78 e3 de fd eb 5e 3b e7 ae f7 eb 6e 3b df ce 9c e1 de 9a e3 de 9c e1 dd f4 e3 9e 9c e1 de 9a e3 5e 9c e1 de 9a e1 ae 7c eb 6d f7 e1 ae f7 e5 ae 37 e7 9e fd e1 de bd e7 9d f7 e7 6e 35 dd dd dd ]&lt;span style=&quot;color:#FFF&quot;&gt;
    &lt;span style=&quot;color:#42a5f5&quot;&gt;base64&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;hex[ c7 bc 67 9b b9 f9 ]&lt;span style=&quot;color:#FFF&quot;&gt;
        &lt;span style=&quot;color:#42a5f5&quot;&gt;unicode-chardet&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;(Windows-1252)&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#ffeb3b&quot;&gt;Ç¼g›¹ù&lt;span style=&quot;color:#FFF&quot;&gt;
    &lt;span style=&quot;color:#78909c&quot;&gt;]base64_urlsafe -&gt; same as base64
&lt;span style=&quot;color:#FFF&quot;&gt;    &lt;span style=&quot;color:#42a5f5&quot;&gt;unicode-chardet&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;(Windows-1251)&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;б®xлЋ7б®xгЮэл^;з®члn;ЯОњбЮљгЮњбЭфгћњбЮљг^њбЮљб®|лmчб®че®7зћэбЮЅзќчзn5ЭЭЭ&lt;span style=&quot;color:#FFF&quot;&gt;
        &lt;span style=&quot;color:#78909c&quot;&gt;]base64 -&gt; (seen before) hex[ c7 bc 67 9b b9 f9 ]
&lt;span style=&quot;color:#FFF&quot;&gt;        &lt;span style=&quot;color:#78909c&quot;&gt;]base64_urlsafe -&gt; (seen before) hex[ c7 bc 67 9b b9 f9 ]
&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#78909c&quot;&gt;]base64_urlsafe -&gt; same as base64
&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#42a5f5&quot;&gt;hex&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;b&#39;JTdCJTIyaGVsbG8lMjIlM0ElMjAlMjJXb3JsZCUyMiU3RA==&#39;&lt;span style=&quot;color:#FFF&quot;&gt;
    &lt;span style=&quot;color:#42a5f5&quot;&gt;base64&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;b&#39;%7B%22hello%22%3A%20%22World%22%7D&#39;&lt;span style=&quot;color:#FFF&quot;&gt;
        &lt;span style=&quot;color:#42a5f5&quot;&gt;url&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;b&#39;{&quot;hello&quot;: &quot;World&quot;}&#39;&lt;span style=&quot;color:#FFF&quot;&gt;
            &lt;span style=&quot;color:#42a5f5&quot;&gt;unicode-utf8&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#ffeb3b&quot;&gt;{&quot;hello&quot;: &quot;World&quot;}&lt;span style=&quot;color:#FFF&quot;&gt;
            &lt;span style=&quot;color:#78909c&quot;&gt;]unicode-chardet(ascii) -&gt; same as unicode-utf8
&lt;span style=&quot;color:#FFF&quot;&gt;        &lt;span style=&quot;color:#42a5f5&quot;&gt;unicode-utf8&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;%7B%22hello%22%3A%20%22World%22%7D&lt;span style=&quot;color:#FFF&quot;&gt;
            &lt;span style=&quot;color:#78909c&quot;&gt;]url -&gt; (seen before) {&quot;hello&quot;: &quot;World&quot;}
&lt;span style=&quot;color:#FFF&quot;&gt;        &lt;span style=&quot;color:#78909c&quot;&gt;]unicode-chardet(ascii) -&gt; same as unicode-utf8
&lt;span style=&quot;color:#FFF&quot;&gt;    &lt;span style=&quot;color:#78909c&quot;&gt;]base64_urlsafe -&gt; same as base64
&lt;span style=&quot;color:#FFF&quot;&gt;    &lt;span style=&quot;color:#42a5f5&quot;&gt;unicode-utf8&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;JTdCJTIyaGVsbG8lMjIlM0ElMjAlMjJXb3JsZCUyMiU3RA==&lt;span style=&quot;color:#FFF&quot;&gt;
        &lt;span style=&quot;color:#78909c&quot;&gt;]base64 -&gt; (seen before) b&#39;%7B%22hello%22%3A%20%22World%22%7D&#39;
&lt;span style=&quot;color:#FFF&quot;&gt;        &lt;span style=&quot;color:#78909c&quot;&gt;]base64_urlsafe -&gt; (seen before) b&#39;%7B%22hello%22%3A%20%22World%22%7D&#39;
&lt;span style=&quot;color:#FFF&quot;&gt;    &lt;span style=&quot;color:#78909c&quot;&gt;]unicode-chardet(ascii) -&gt; same as unicode-utf8
&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#42a5f5&quot;&gt;unicode-utf8&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#1976d2&quot;&gt;&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#4caf50&quot;&gt; -&gt; &lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#8bc34a&quot;&gt;4a5464434a544979614756736247386c4d6a496c4d30456c4d6a416c4d6a4a5862334a735a4355794d69553352413d3d&lt;span style=&quot;color:#FFF&quot;&gt;
    &lt;span style=&quot;color:#78909c&quot;&gt;]base64 -&gt; (seen before) hex[ e1 ae 78 eb 8e 37 e1 ae 78 e3 de fd eb 5e 3b e7 ae f7 eb 6e 3b df ce 9c e1 de 9a e3 de 9c e1 dd f4 e3 9e 9c e1 de 9a e3 5e 9c e1 de 9a e1 ae 7c eb 6d f7 e1 ae f7 e5 ae 37 e7 9e fd e1 de bd e7 9d f7 e7 6e 35 dd dd dd ]
&lt;span style=&quot;color:#FFF&quot;&gt;    &lt;span style=&quot;color:#78909c&quot;&gt;]base64_urlsafe -&gt; (seen before) hex[ e1 ae 78 eb 8e 37 e1 ae 78 e3 de fd eb 5e 3b e7 ae f7 eb 6e 3b df ce 9c e1 de 9a e3 de 9c e1 dd f4 e3 9e 9c e1 de 9a e3 5e 9c e1 de 9a e1 ae 7c eb 6d f7 e1 ae f7 e5 ae 37 e7 9e fd e1 de bd e7 9d f7 e7 6e 35 dd dd dd ]
&lt;span style=&quot;color:#FFF&quot;&gt;    &lt;span style=&quot;color:#78909c&quot;&gt;]hex -&gt; (seen before) b&#39;JTdCJTIyaGVsbG8lMjIlM0ElMjAlMjJXb3JsZCUyMiU3RA==&#39;
&lt;span style=&quot;color:#FFF&quot;&gt;&lt;span style=&quot;color:#78909c&quot;&gt;]unicode-chardet(ascii) -&gt; same as unicode-utf8
&lt;span style=&quot;color:#FFF&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;p&gt;As you can see, the program tried various encodings recursively, which lead to two possible solutions. The first one appears to be a false positive, leading to some garbled text. The second one definitely looks more promising, applying &lt;code&gt;hex -&amp;gt; base64 -&amp;gt; url -&amp;gt; utf-8&lt;/code&gt; decoding to obtain some JSON string.&lt;/p&gt;&lt;p&gt;Duplicate paths using different encodings have also been detected and skipped.&lt;/p&gt;&lt;h3 id=&quot;as-a-python-library&quot;&gt;As a Python library&lt;/h3&gt;&lt;pre class=&quot;language-python&quot; tabindex=&quot;0&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token keyword&quot;&gt;from&lt;/span&gt; multidecoder &lt;span class=&quot;token keyword&quot;&gt;import&lt;/span&gt; multidecode&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; display_result

results &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; multidecode&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;text&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; max_depth&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
display_result&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;results&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; sys&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;stdout&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;how-does-it-work&quot;&gt;How does it work?&lt;/h2&gt;&lt;p&gt;The &lt;code&gt;multidecode()&lt;/code&gt; function will go through a list of decoders, attempting the decode the input text with each one in turn.&lt;/p&gt;&lt;p&gt;If the decoder output is equal to the input, or the decoder errors out, the &amp;quot;branch&amp;quot; is going to be skipped.&lt;/p&gt;&lt;p&gt;Otherwise, a &lt;code&gt;Result&lt;/code&gt; is yielded for each successful decoding operation, with the following fields:&lt;/p&gt;&lt;ul&gt;&lt;li&gt;&lt;code&gt;decoder_id&lt;/code&gt;: internal identifier for the decoder&lt;/li&gt;&lt;li&gt;&lt;code&gt;value&lt;/code&gt;: return value from the decoder&lt;/li&gt;&lt;li&gt;&lt;code&gt;args&lt;/code&gt;: list of strings, used by decoders to return extra information about the decode process. A common example is the &lt;code&gt;chardet&lt;/code&gt; decoder, which will attempt to automatically guess what unicode encoding was used to encode some text.&lt;/li&gt;&lt;li&gt;&lt;code&gt;is_new_path&lt;/code&gt;: flag indicating whether this result &lt;code&gt;value&lt;/code&gt; has been seen before. Display and search algorithm might want to use this to avoid descending the same branch twice unnecessarily, or possibly ending in an infinite loop.&lt;/li&gt;&lt;li&gt;&lt;code&gt;sub_results&lt;/code&gt;: iterator of &lt;code&gt;Result&lt;/code&gt;s obtained by recursively decoding this result.&lt;/li&gt;&lt;/ul&gt;</content>
  </entry>
  <entry>
    <title>Hello 11ty</title>
    <link href="https://example.com/blog/first-post/" />
    <updated>2025-12-19T00:00:00Z</updated>
    <id>https://example.com/blog/first-post/</id>
    <content type="html">&lt;p&gt;Trying to revamp this blog, again.&lt;/p&gt;&lt;p&gt;This time using &lt;a href=&quot;https://www.11ty.dev/&quot;&gt;11ty&lt;/a&gt;. Hoping this will prove a more stable and future-proof solution than previous choices of site generator.&lt;/p&gt;&lt;h2 id=&quot;plans-for-the-future&quot;&gt;Plans for the future&lt;/h2&gt;&lt;p&gt;I&#39;m going to be writing (mostly) about:&lt;/p&gt;&lt;ul&gt;&lt;li&gt;Software architecture&lt;/li&gt;&lt;li&gt;Open Source projects&lt;/li&gt;&lt;li&gt;Computer security&lt;/li&gt;&lt;li&gt;Technology and other cool stuff&lt;/li&gt;&lt;/ul&gt;</content>
  </entry>
</feed>