Skip to main content Logo

Quis custodiet ipsos custodes?
Home | About | All pages | Cluster Status | RSS Feed | Gopher

Bash bits: split a file in blocks and do something with each block

Published: 02-09-2019 | Author: Remy van Elst | Text only version of this article

❗ This post is over two years old. It may no longer be up to date. Opinions may have changed.

Table of Contents

Bash Bits are small examples, tips and tutorials for the bash shell. This bash bit shows you how to split a file into blocks (multiline) and do something with each block. This can be used for certificate chains or other files which have multiline blocks.

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

All Bash Bits can be found using this link

Splitting a file in blocks

Most actions on text based files like csv files allow you to split a line into multiple parts seperated by, for example, a comma. If you have a file which has blocks spanning multiple lines, it's harder to find a good guide how to do something with that. This guide tries to be as clear as possible.

The file example I'm using is a certificate chain file. You've probably seen one of those, it has a few certificates in it, truncated example:


This file has clear seperate "entities", namely the certificates. Those entities are seperated by the lines -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----. When I'm referring to a block in this guide, I'm talking about that one part. In this example that's one certificate.

It could also be that you have a long list file where entities are seperated between blocks, for example a combined .csv file you want to split up.

If you need a certificate chain file, fire up your favorite search engine and search for one. An example can be found here, scroll down to Certificate Chain, open it and save it.

My file is named test.crt, replace it in the commands with your filename.

The below command splits up the file into blocks and prints the first two lines of a block to the shell:

OLDIFS=$IFS; IFS=';' blocks=$(sed -n '/-----BEGIN /,/-----END/ {/-----BEGIN / s/^/\;/; p}'  test.crt);
for block in ${blocks#;}; do 
    echo $block | head -n 2
    echo "==== SEPERATOR ===="
done; IFS=$OLDIFS 

Example output:

==== SEPERATOR ====
==== SEPERATOR ====
==== SEPERATOR ====

Here's a short explanation of what the loop does:

If your file has semicolons in it, you must change the seperator in the commands. It could be a colon, or another unused character.

If your file is seperated by another text, the format for the split is as follows:


The top split line must be there twice.

Doing something with the seperate blocks

Printing out the contents of the blocks isn't super usefull, you could have just done that with the entire file. Maybe you want to split the file up into seperate files. Let's add a counter and split the chain up into seperate files:

COUNTER=1; OLDIFS=$IFS; IFS=';' blocks=$(sed -n '/-----BEGIN /,/-----END/ {/-----BEGIN / s/^/\;/; p}'  test.crt);
for block in ${blocks#;}; do 
    echo "file $COUNTER"
    echo $block > cert-$COUNTER.crt;
done; IFS=$OLDIFS 

You now have three seperate files:

$ ls cert*
-rw-rw-r-- 1 remy remy 2.2K Sep  2 11:44 cert-1.crt
-rw-rw-r-- 1 remy remy 1.8K Sep  2 11:44 cert-2.crt
-rw-rw-r-- 1 remy remy 1.4K Sep  2 11:44 cert-3.crt

Or maybe you want to print the common name of each certificate:

OLDIFS=$IFS; IFS=';' blocks=$(sed -n '/-----BEGIN /,/-----END/ {/-----BEGIN / s/^/\;/; p}'  test.crt);
for block in ${blocks#;}; do 
    echo $block | openssl x509 -noout -subject -in -
done; IFS=$OLDIFS 

Example output:

subject=C = NL, L = Den Haag, O = Koninklijke Bibliotheek, OU = ICT, CN =
subject=C = NL, ST = Noord-Holland, L = Amsterdam, O = TERENA, CN = TERENA SSL CA 3
subject=C = US, O = DigiCert Inc, OU =, CN = DigiCert Assured ID Root CA
Tags: array , awk , bash , bash-bits , ifs , openssl , sed , shell , split , tutorials