Raymii.org

Quis custodiet ipsos custodes?
Home | About | All pages | Cluster Status | RSS Feed

Bash bits: split a file in blocks and do something with each block

Published: 02-09-2019 | Author: Remy van Elst | Text only version of this article

❗ This post is over four years old. It may no longer be up to date. Opinions may have changed.

Splitting a file in blocks
Doing something with the seperate blocks

Bash Bits are small examples, tips and tutorials for the bash shell. This bash bit shows you how to split a file into blocks (multiline) and do something with each block. This can be used for certificate chains or other files which have multiline blocks.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

All Bash Bits can be found using this link

Splitting a file in blocks

Most actions on text based files like csv files allow you to split a line into multiple parts seperated by, for example, a comma. If you have a file which has blocks spanning multiple lines, it's harder to find a good guide how to do something with that. This guide tries to be as clear as possible.

The file example I'm using is a certificate chain file. You've probably seen one of those, it has a few certificates in it, truncated example:

-----BEGIN CERTIFICATE-----
MIIGO[...]grxckatBjE6CayMDaIHXKgI8oNu/snqhxGfcnrQyS6iu1libVL9VdsNFgCUyBcgl
dtlM9GjvT6JtMVYW
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIE+zCCA+OgAwIBA[...]hS+1NGClXwmgmkMd1L8tRNaN2v11y18WoA5hwnA9Ng==
-----END CERTIFICATE-----

This file has clear seperate "entities", namely the certificates. Those entities are seperated by the lines -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----. When I'm referring to a block in this guide, I'm talking about that one part. In this example that's one certificate.

It could also be that you have a long list file where entities are seperated between blocks, for example a combined .csv file you want to split up.

If you need a certificate chain file, fire up your favorite search engine and search for one. An example can be found here, scroll down to Certificate Chain, open it and save it.

My file is named test.crt, replace it in the commands with your filename.

The below command splits up the file into blocks and prints the first two lines of a block to the shell:

OLDIFS=$IFS; IFS=';' blocks=$(sed -n '/-----BEGIN /,/-----END/ {/-----BEGIN / s/^/\;/; p}'  test.crt);
for block in ${blocks#;}; do 
    echo $block | head -n 2
    echo "==== SEPERATOR ===="
done; IFS=$OLDIFS

Example output:

Here's a short explanation of what the loop does:

First it saves the old IFS, the internal field seperator. The IFS refers to a variable which defines the character or characters used to separate a pattern into tokens for some operations, mostly a newline, space or tab.
Second, the IFS is set to a semicolon (;)
The file is split up into blocks using sed, and between each block a semicolon is inserted.
Using bash syntax, the blocks are put into a variable in a foreach loop, split on the semicolon
Inside the loop an action is done
The IFS is restored to what it was.

If your file has semicolons in it, you must change the seperator in the commands. It could be a colon, or another unused character.

If your file is seperated by another text, the format for the split is as follows:

sed -n '/TOP SPLIT LINE/,/BOTTOM SPLIT LINE/ {/TOP SPLIT LINE/ s/^/\;/; p

The top split line must be there twice.

Doing something with the seperate blocks

Printing out the contents of the blocks isn't super usefull, you could have just done that with the entire file. Maybe you want to split the file up into seperate files. Let's add a counter and split the chain up into seperate files:

COUNTER=1; OLDIFS=$IFS; IFS=';' blocks=$(sed -n '/-----BEGIN /,/-----END/ {/-----BEGIN / s/^/\;/; p}'  test.crt);
for block in ${blocks#;}; do 
    echo "file $COUNTER"
    echo $block > cert-$COUNTER.crt;
    COUNTER=$((COUNTER +1))
done; IFS=$OLDIFS

You now have three seperate files:

$ ls cert*
-rw-rw-r-- 1 remy remy 2.2K Sep  2 11:44 cert-1.crt
-rw-rw-r-- 1 remy remy 1.8K Sep  2 11:44 cert-2.crt
-rw-rw-r-- 1 remy remy 1.4K Sep  2 11:44 cert-3.crt

Or maybe you want to print the common name of each certificate:

OLDIFS=$IFS; IFS=';' blocks=$(sed -n '/-----BEGIN /,/-----END/ {/-----BEGIN / s/^/\;/; p}'  test.crt);
for block in ${blocks#;}; do 
    echo $block | openssl x509 -noout -subject -in -
done; IFS=$OLDIFS

Example output:

subject=C = NL, L = Den Haag, O = Koninklijke Bibliotheek, OU = ICT, CN = www.bibliotheek.nl
subject=C = NL, ST = Noord-Holland, L = Amsterdam, O = TERENA, CN = TERENA SSL CA 3
subject=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Assured ID Root CA

Tags: array , awk , bash , bash-bits , ifs , openssl , sed , shell , split , tutorials

Bash bits: split a file in blocks and do something with each block

Table of Contents

Splitting a file in blocks

Doing something with the seperate blocks