Difficulty: Low
When it comes to sending Bitcoin transactions of any kind, I try to pay the smallest fee possible. With the current popularity of inscriptions the fee market has changed. Apparently I’m still adjusting to the change and recently made a mistake in picking my transaction fee when opening a lightning channel with my Core Lightning node. The result was a lightning channel open transaction that would never have confirmed.
Under normal circumstances this shouldn’t really happen. If you allow Core Lightning to pick the fee for you the application will choose a high enough fee that your transaction will confirm quickly. However as noted above, I’m pretty stingy with my sats! On a Sunday evening I broadcast a channel open transaction with a 2 sat/vByte fee and went to bed.
When I woke up the next morning I was surprised to see that the transaction had not confirmed. As it was the beginning of the work week I realized that it would be until the following weekend that the transaction would have another reasonable chance to confirm. The following weekend came and went and my transaction was still stuck in the mempool. I wasn’t sure what would happen if the channel open was still in progress and my transaction aged out of the mempool after two weeks and I decided that I didn’t want to find out. I created the channel with an entire single UTXO so I didn’t have a change output and therefore CPFP was not an option. I figured I would have to find a way to double spend myself.
The Core Lightning team provides excellent documentation. They do provide good instructions to deal with a situation like I found myself in:
https://docs.corelightning.org/docs/faq#how-to-forget-about-a-channel
Unfortunately I didn’t find these instructions initially. I would end up finding this document later and it did prove helpful for the steps after the double spend. However without the official Core Lightning double spend instructions I set off on my own to figure out a way to double spend myself.
I didn’t find a way to double spend from inside the Core Lightning on chain wallet natively. Frankly I’m not sure if any wallet supports that function. If a wallet knows it has broadcast a transaction it would make sense that it wouldn’t try to spend the same UTXO twice. With this in mind I figured I would have to extract the Core Lightning on chain wallet keys and import them into another wallet.
The Core Lightning team provides an hsmtool that can perform functions on the hsm_secret file which contains the master private key. One of the functions provided by the hsmtool sounded promising. This was the dumponchaindescriptors function. Unfortunately this only provides the public keys and is intended to be used with a watch only wallet. This is a useful function but not what I was looking for.
Eventually I came across this excellent tool which would do exactly what I was looking for:
https://github.com/jb55/clightning-dumpkeys
As a side note, the author of this tool is the now pretty well know jb55 of Nostr client Damus fame. In the past he was involved with the Core Lightning project. I must be nice to be such a skilled developer!
The instructions on the GitHub page are fairly easy to follow. The tool does have to be compiled manually. I found that I was unable to compile the tool on my Arch Linux installation but I did successfully compile it on Ubuntu 20.04. I moved my hsm_secret file to the Ubuntu 20.04 installation and ran the clightning-dumpkeys program on it.
The program will dump your xprv which is needed for inport the wallet into Bitcoin Core. The output from clightning-dumpkeys looks like this:
xprv9s21ZrQH143K2W9UqiTJL3qndXzRkEuzAoNwd6VLRu2VDtX6tACVz9q8BzDmJDkB6c6QQnfEWWvLtxB9M68XVpmcJCQodFrr843paWAXGeD root private
xpub661MyMwAqRbcEzDwwjzJhBnXBZpv9hdqY2JYRUtwzEZU6grFRhWkXx9c3HJ4EKR1Nvdwf5U3VoekstoKSKjfcJYhRrhMYeEzZzu2h7uZAQX root public
xprv9wPk6zRy29x8BLXUHWh8ssqf89xQWtrZeLUgEXyUKSqNdUJhXTJsMtdLCexXxhgxPYUFpQvUhz2WAyhh3uCXSLedbHkvX7jTnX4xQofjdbA extended private
combo(xprv9wPk6zRy29x8BLXUHWh8ssqf89xQWtrZeLUgEXyUKSqNdUJhXTJsMtdLCexXxhgxPYUFpQvUhz2WAyhh3uCXSLedbHkvX7jTnX4xQofjdbA/*)#u4tc9nwu extended private descriptor
xpub6AP6WVxrrXWRPpbwPYE9F1nPgBntvMaR1ZQH2vP5snNMWGdr4zd7ugwp3wukcTUxKu2rLCN9VBQAW3xioATnEWjZvQpx9cybj1jztJHJyp7 extended public
combo(xpub6AP6WVxrrXWRPpbwPYE9F1nPgBntvMaR1ZQH2vP5snNMWGdr4zd7ugwp3wukcTUxKu2rLCN9VBQAW3xioATnEWjZvQpx9cybj1jztJHJyp7/*)#f64dm6yh extended public descriptorThe next step was to create the descriptors.json file as follows:
[
{
"desc": "combo(xprv9wPk6zRy29x8BLXUHWh8ssqf89xQWtrZeLUgEXyUKSqNdUJhXTJsMtdLCexXxhgxPYUFpQvUhz2WAyhh3uCXSLedbHkvX7jTnX4xQofjdbA/*)#u4tc9nwu",
"keypool": true,
"timestamp": "now",
"watchonly": false,
"internal": false,
"range": [0, 1000]
}
]Importantly to import this wallet descriptor into Bitcoin I had to set keypool to true and watchonly to false which deviates from jb55’s example on the GitHub page. This format of the descriptors.json file can be used to import the Core Lightning wallet into Bitcoin Core in a way that the funds in the on chain wallet can be spent.
The next step is the actual import into Bitcoin Core. This can be accomplished with bitcoin-cli run from a terminal window. These commands will not work from the Bitcoin-QT console. You will need to have a running and fully synced node. Then run these commands from a terminal window:
$ bitcoin-cli -named createwallet wallet_name=core-lightning disable_private_keys=false descriptors=true load_on_startup=true
$ bitcoin-cli -rpcwallet=core-lightning importdescriptors "$(cat descriptors.json)"
$ bitcoin-cli -rpcwallet=core-lightning rescanblockchain 503500Again, I deviated slightly from jb55’s GitHub instructions since I wanted to make the funds in the newly imported wallet spendable. I set disable_private_keys to false.
One other thing to note. You can set rescanblockchain to any block number prior to the creation of your Core Lightning node.
Success! After the rescan completed I could see the transaction history for the on chain wallet of my Core Lightning node. I could also see that my stuck UTXO was in the wallet and spendable. I generated a new address on my Core Lightning node and sent a new transaction, this time with a sufficient fee. A bit later I could see that the transaction had confirmed. The double spend was complete and my funds were safe!
There was only one small problem. My node still had the channel that had failed to open listed as pending. At this point I followed the steps in the document linked above.
The Core Lightning team recommends that the unopened channel be force-closed. I ran this command where $PEERID is my attempted channel partner’s node public key:
$ lightning-cli close $PEERIDThis command just hung and eventually I CTRL-C. I suspect the command hung because there was no open channel to actually force close.
For the final step, I had to tell my node to forget the channel. I ran this command where $NODEID was once again my attempted channel partner’s node public key:
$ lightning-cli dev-forget-channel $NODEIDThe pending channel was now gone from my node. Everything was as it was before this fiasco started and nothing was lost except for a handful of sats spent on the double spend transaction fee.
One important thing to note. For the dev-forget-channel command to be available I needed to recompile Core Lightning with the developer option enabled:
./configure --enable-developerSince I believe in finishing what I start I felt I needed to try opening a channel to the same lightning node that I had attempted to open a chennel with when my transaction got stuck in the mempool. Since I sent the double spend transaction back to my Core Lightning node I was ready to try the channel open again. This time I sent my channel opening transaction with a sufficiently high fee, the transaction confirmed, and the channel opened successfully.
Hopefully you won’t find yourself in this situation but if you do, don’t panic. There are quality tools out there to help you retrieve your Bitcoin from the now busy mempool.