Replace values in CSV file

A two-pass script that uses Miller to create a mapping object and then apply it in a replacement loop.

Created 2024 by Pontus Sundén.

Input from fancy command line flags

#! /bin/zsh -

zmodload zsh/zutil
zparseopts -A ARGUMENTS -input: -mapping: -column: -separator:

# data file
input=${ARGUMENTS[--input]:-input.csv} 

# csv file with two columns in this order: "old, new" (only order is important)
raw_map=${ARGUMENTS[--mapping]:-[]}

# name of columns to map on
column=${ARGUMENTS[--column]:-column}

# the separator
separator=${ARGUMENTS[--separator]:-,}

Syntax:

variable=${ARGUMENTS[--flag]:-default}

Example command:

script.sh --flag "foo"

will set variable to "foo". If flag is absent, it's set to "default"

Utility file

mapping="_mapping_used_$raw_map.mlr"

mlr='begin {\n'
mlr+='@separator = "'$separator'";\n'
mlr+='@column = "'$column'";\n'
mlr+='@mapping = '$(mlr --c2j filter 'is_not_empty($[[[1]]])' then put 'old=$[[[1]]]; new=$[[[2]]]; unset $[[[1]]]; unset $[[[1]]]; $old=old; $new=new' "$raw_map")
mlr+=';\n}'
printf "$mlr" > "$mapping"

Main command doing the mapping

Loading variables from utility file:

@column : the column to map on
@separator : the separator to use if the field is multi-value
@mapping : an array of objects with two elements: {"old": "old_value", "new": "new value"}

mlr --csv \
put -f "$mapping" -e '
  updated = [];
  for ( n,old_value in splita($[@column], @separator) ) {
    new_value = old_value;
    for ( mapping_pair in @mapping ) {
      if (old_value == mapping_pair.old) {
        new_value = mapping_pair.new;
      }
    }
    updated = append(updated, new_value);
  }
  $[@column] = joinv(updated, @separator);
' "$input"

psu/csv-translate-with-mapping.sh.md

Select an option