grep speedup trickery

I polished my parallel grep version. When I started it in 2012, multicore + SSD setups were not that common. Today, lot of storage is on flash or SSD, so you can benefit from parallel grepping by a factor of 3 or more (depending on amount of CPU cores). Just check out the link; it will also contain some timed runs to underline the statements. I also noticed that my previous git singing key expired, so I will need to resign the repos with my new GPG key (already uploaded) over time.

Update: I added a new branch to the repo to again double the speed by an dedicated nftw() + readdir() implementation thats parallelized and recursive at the same time! If you enjoy brainfucks, give it a try!

pam_python trickery

I made a writeup about a pam_python issue
here (CVE-2019-16729 incomplete fix). pam_python is not
widely deployed, but some more fancy authentication
frameworks like face recognition on Linux seem to
require it.

D'oH! no-IPv6-workaround trickery

I added a workaround to my harddns DoH project. Some browsers accept AAAA records and prefer them over A records, even when there is no IPv6 connectivity. Inside the NSS module, it is hard to distinguish these cases and its certainly not our task to check for IPv6 connectivity during DNS resolves. Therefore, I added a nss_aaaa config flag, which needs to be enabled if you want the NSS module to lookup AAAA records for certain gethostbyname() calls. Certain - because within various incarnations like gethostbyname2(), getaddrinfo() etc., some of these functions lead to NSS ‘backend’-functions which can make the case by checking an address-family parameter. Some functions can’t and will use the nss_aaaa helper flag from the harddns config. It is disabled by default, so if you use harddns via the NSS config, you need to enable it, if you have IPv6 connectivity and want to profit from it. The DoH proxy server works as before, since its just translating between DNS and DoH back and forth.

IPv6 massbind trickery

Some more open-source community love for handling of lot of ip6 addresses on a single interface, to pick and dispose per connection for example to evade address based statistics:
here.

I also cleaned up this' blog link-list to free it
from dangling links and adding some new which I found
interesting to read.

Hey folks, if you are wondering what I may be doing, I am still
splitting my time in coding and reading of other ppls code.
Here's some results.

I analyzed quite some DoH "solutions" for mobile and desktop (browsers),
and I can tell you that most of them are really bloated. What do you think,
how many TCP/IP stacks you can stack on an Android phone? If you think
you may be just running one or two: good chances that you may be wrong.

As it turned out even malware authors jump onto the DoH train, but
nevertheless the experts are discussing whether its really DoH or not.

OTOH, I used the time to add some new stuff to my own DoH
repo.

The fancy new features are:

 * Support for BSD and OSX
 * caching daemon
 * support for rfc8484
 * supports all major DoH providers out of the box

Sure, nothing that the big tech companies offer you is for free, not
even DoH! So apparently even your DNS requests are of interest. So you
have to decide yourself whether you accept DNS redirects by your gov
to inject some malware, or allow tech companies to play big data
with your lookups.

After all, porting harddns or any other non-trivial code to BSD,
you gotta love -pedantic. Still, BSD has its special needs and
its always good to know.

new opmsg trickery

Theres two news to the opmsg:

First: I added brainkey personas, a similar thing to
BTC brainwallets. If you use these, you do not need
to send your pubkeys around anymore. It's an experimental
feature. The key is derived via PBKDF2 function with HMAC
with a lot of iterations and salt, so to prevent large
scale (or huge-scale) dictionary attacks. True, I could have used
memory-hard functions for that but a) Argon2 or friends
are not yet part of OpenSSL and b) if I understand correctly,
you can still use it on massive paralleled ASCICs if you
accept additional computation cost for re-creating matrices
on the fly. So a+b let me chose PBKDF2 over Argon2.
Brainkey personas are also deniable personas, which underlines
the concept of linked/deniable personas.

Second: you can im/export your persona keys as QR codes
now, to make it more convenient to exchange pubkeys
due to ubiquitous (what a word!) dumb phones that can send around screenshots.
Its inside its own git repo that you can find here.
Most of the effort was finding and integrating small
footprint QR encode/decoding libraries.

DoH trickery

Made a posting about DoH integration and latency measurement.

Bitcoin Census trickery

I wish I could have posted this on my gh-pages blog site,
but apparently I do not have sufficient skills to embed
script tags into their markdown rendering.
So, in order to show some fancy maps I have to use this blog again.
Stay tuned, and regularily check both of my blogs for updates;
because gh-pages is better for embedded code snippets while
this seems better for embedding HTML.

I published my BTC mapping engine hoschi.

It will crawl through the BTC p2p network, fetching all
addresses it can. The obtained raw data may be used for
further analysis like mapping it to geo locations or building
graphs of the connected nodes in order to spot unusual network
setups or to locate mining farms.

For instance, scanning through the BTC testnet, yields this map:



The cool thing is that github will automatically render
geojson files and also cluster the points for me.
The map for the BTC main network is too large to be rendered
on gh.
Nevertheless, I was actually interested in big nodes:


which are connected to more than 1000 other BTC nodes in
the network. In my census, I found ~272k mainnet BTC nodes,
~1k of which are big-nodes. These could either be just
long running nodes so that a lot of other nodes have had
the chance to connect to it, or nodes located nearby
mining farms in order to distribute mined blocks quickly
across the p2p network. However, it may also be possible
that mining farms use their own BTC software not answering
to getaddr requests.
Some of the big nodes seem to be special indeed, as whois lookups showed weird registrar naming entries such as Data Bureau.

Other interesting occurences were nodes that tried to
join testnet and mainnet, heavily multihomed nodes, or
nodes distributing multicast or otherwise special IP addresses. According to my census, IPv6 is already much more widespread than
one would think. 22% of the BTC nodes were IPv6.

QDNS rebinding trickery

I am trying to do some posts via my gh pages.
I will cross-link my posts on this blog in future,
so you don't need to update your bookmarks.

Both blogs will stay free of advertisement or
influencer shit in 2018 as before, except the usual C++11
(now C++17) attitude towards Rust coders. :p

harddns usability

I updated harddns to make use of standard TLS certificate
checking. Usage of pinned certs is now optional. If they
exist inside /etc/harddns/pinned, they have to match
in order for the resovle to succeed. If there don't exist
any pinned certs, the common X509 chain verification
is applied.
This should make harddns much more usable, as
there are plenty of X509's from google endpoints
which may appear if the resolver web API is contacted.