Csdrhrt

I am using IPSet to managed tens of thousands of IPv4 CIDR/netmask ranges that then get linked to IPTables rules. This setup is working great, but I would like to get a good, high-level overview count of the IP host addresses IPSet acts on for client reporting purposes.

The IPSet entry formatting is consistently like this:

123.456.0.0/16 timeout 86400

So I can grep on the lines that have timeout to get values to act on the CIDR/netmask ranges that the entry contains.

For example, if I save the IPSet output (via ipset -L -n > ipset-20181228.txt) to a text file named ipset-20181228.txt and then run a combination of grep and wc -l like this:

grep  "timeout" ipset-20181228.txt  | wc -l

I get a count of 39,000+ items which equate to 39,000+ CIDR/netmask ranges. But that is (of course) only counting the CIDR/netmask ranges and not full counts of IP host addresses in that range.

I attempted to use prips (which expands CIDR/netmask values to actual IP addresses in Bash) with grep to cull out only items with CIDR/netmask ranges like this:

grep -oE '(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])/([0-9]{1,2})' ipset-20181228.txt | awk 'NF { system( "prips " $0)  }' | wc -l

And after a whopping 20 to 30 minutes (!!!) on my 2018 MacBook Air (with the fans kicking in), the count I got was 736,000,000+ which is what I am going for… But 20 to 30 minutes is way too long. I want this to be as scriptable and non-intrusive as possible, and can’t trust a command like that to run on a production server without eating up resources; I mean look at how it behaves on my local 2018 MacBook Air development setup.

Is there any way to just calculate the CIDR/netmask range count based on simply the CIDR/netmask value? I am hoping there is just some command line tool—or option in existing tools I am using—that I am unaware of that can help.

If your grep command outputs lines like 123.456.0.0/16, then you need to pipe them to

awk -F / '{ count[$2]++ } END { for (mask in count) total+=count[mask]*2^(32-mask); print total }'

The command only extracts masks (i.e. what is after /) and counts occurrences of each mask. At the end the number of hosts is calculated for each encountered mask (2^(32-mask)), multiplied by the number of occurrences and summed up.

Notes:

• No sanity check is performed. E.g. input like 1.2.3.4/40 will be accepted, non-integer output will be calculated. Improve your preliminary grep filter if needed.


• Each range independently contributes to the total number. If your ranges overlap then you will get an inflated result (I think your try with prips was no better in this).

I thought something like this would work since the original poster is running grep to get the CIDR from lines with timeout:

awk -F'[ /]' '/timeout/ {hosts+=2^(32-$2)};ENDFILE{print "Hosts number in "FILENAME": "hosts;total+=hosts;hosts=0};END {print "Total: "total}' ipset*.txt

EDIT- The awk program above runs ok only with GNU awk. ENDFILE is a GNU extension.

I guess BSD awk ignores ENDFILE and runs that section if it was part of the main program section.

This is compatible with GNU and BSD awk.

awk -F'[ /]' '{if (filename != FILENAME) hosts=0};/timeout/ {hosts+=2^(32-$2)};{filename=FILENAME;file_total[filename]=hosts};END{for (i in file_total) {print "Hosts number in "i": "file_total[i];total+=file_total[i]};{print "Total: "total};}' ipset*.txt