Csdrhrt

I have several big files with some measurements.

It looks this way:

N 12344;PE 9.9999999;...

#S 0 0 31 44 75 130 165 196...

#S_+ "2 5 2 3 3 1 1 2 3 1 2 2...



N 12345;PE 9.9999999;...

#S 0 0 34 57 84 133 152...

#S_+ "1 0 1 1 2 3 0 0 0...



N 12346;PE 9.9999999;...

#S 0 0 31 44 73 140 169...

#S_+ "3 3 4 0 0 2 1 2 4...



N 25104;PE 9.9999999;...

#S 0 0 36 52 102 108 145...

#S_+ "1 1 0 1 0 0 3 0 1...



N 25105;PE 9.9999999;...

#S 0 0 32 58 88 130 143...

Sample is here:
http://pasted.co/d9806b7c4

The file is much bigger but I replaced part of the data with "..." to make it shorter.

I need to somehow replace the line ends before "#S" - in fact simply merge the "N" line with the following two ones into one line (or with the following three ones so I can get rid of the blank lines). Expect output like this:

N 12344;PE 9.9999999; #S 0 0 31 44 75 130 165 196 #S_+ "2 5 2 3 3 1 1 2 3 1 2 2...

N 12345;PE 9.9999999; #S 0 0 34 57 84 133 152 #S_+ "1 0 1 1 2 3 0 0 0...

N 12346;PE 9.9999999; #S 0 0 31 44 73 140 169 #S_+ "3 3 4 0 0 2 1 2 4...

N 25104;PE 9.9999999; #S 0 0 36 52 102 108 145 #S_+ "1 1 0 1 0 0 3 0 1...

N 25105;PE 9.9999999; #S 0 0 32 58 88 130 143...

Is this possible to achieve using some command-line utility in linux?

My knowledge is quite limited in this area so I would appreciate any help.

thanks

With sed:

sed -z -e 's/n#S/ #S/g' -e 's/nN /N /g' data

In slow-mo:

• 
  -z makes sed consider the file as a single line (so the line ends are plain characters)


• 
  's/n#S/#S/g' replaces all LF's occurring just before a #S by a space


• 
  -e 's/nN /N /g' replaces all LFs before N (ie, the blank lines)

With paste (this requires to always have groups of 4 lines):

 paste -s -d '   n' data

In slo-mo:

• 
  paste -s concatenates the lines from the file


• 
  -d specifies characters to be inserted as delimiters. When there are several characters, they are used in a round-robin fashion, so with 3 spaces and a LF:
  
  
  
  
  • the first space is used on the first splice (N to #S),
  
  
  • the second space is used on the second splice (#S to #S),
  
  
  • the third space is used on the thrid splice (#S to blank line),
  
  
  • the last delimiter, a LF, is used on the fourth splice (blank line to N)
  
  
  • and the cycle repeats for the next 4 lines.
  
  
  
  

This is a portable solution with POSIX sed, implementing the following rules:

• empty lines shall be deleted;


• any line starting with #S shall be merged with the previous non-empty line, with a single space character between them, unless there is no previous non-empty line.

The code:

<data sed '/^$/ d; :start; N; s/n$//; t start; s/n#S/ #S/; t start; P; D'

The same with comments (still working code):

<data sed '

  /^$/ d      # If empty line read, delete it and start a new cycle.

  :start      # A label.

  N           # Read additional line, there are now two lines in the pattern space.

  s/n$//     # If the second line is empty, replace the newline with nothing.

  t start     # If the above replacement occurred, go to start (to add another line).

              # Otherwise

  s/n#S/ #S/ # if the second line starts with #S, replace the newline with space.

  t start     # If the above replacement occurred, go to start (to add another line).

              # Otherwise

              # (i.e when non-empty line not starting with #S occurred)

  P           # print the pattern space up to the first newline and...

  D           # delete the initial segment of the pattern space

              # through the first newline (i.e. everything just printed),

              # and start the next cycle with the resultant pattern space

              # and without reading any new input

              # (in our case the new input will be explicitly read by N then).

  '

Note the solution uses sed pattern space to accumulate many input lines. This remark applies:

The pattern and hold spaces shall each be able to hold at least 8192 bytes.

Just before the P command the pattern space holds one (relatively long) line meant to be printed and a single (relatively short) input line, plus a newline in between. Obviously it depends on your data, whether or not such structure exceeds 8192 bytes at some point. If it does, some sed implementations may fail.

Using Perl:

perl -0 -ape 's/R(?=RN|#)/ /g' file.txt

N 12344;PE 9.9999999;... #S 0 0 31 44 75 130 165 196... #S_+ "2 5 2 3 3 1 1 2 3 1 2 2...

N 12345;PE 9.9999999;... #S 0 0 34 57 84 133 152... #S_+ "1 0 1 1 2 3 0 0 0...

N 12346;PE 9.9999999;... #S 0 0 31 44 73 140 169... #S_+ "3 3 4 0 0 2 1 2 4...

N 25104;PE 9.9999999;... #S 0 0 36 52 102 108 145... #S_+ "1 1 0 1 0 0 3 0 1...

N 25105;PE 9.9999999;... #S 0 0 32 58 88 130 143...

Regex explain:

s/              : substitute

    R          : any kind of line break (ie. r, n, rn)

    (?=         : positive lookahead, zero-length assertion that make sure we have after

        RN     : a line break followed by letter N

      |         : OR

        #       : # character

    )           : end lookahead

/ /g            : replace with a space, global

awk (gawk ^[1])

As usually other than sed you can use awk (and in many different ways...)

awk 'ORS=" "; NR % 4 == 0 && ORS="n" ' data

where

• 
  ORS=" " fixes the output record separator, by default a newline, to a space (you can change)


• 
  NR % 4 == 0 && ORS="n" each 4th line it fixes back to the newline n
  


• If nothing else is specified awk prints the full line


• 
  data is your data file.

If you want you can use regex as in sed (in a similar way).

A format check version with awk

Even if not requested, you may want to manage a truncated file eliminating the corrupted output line and generating an error and an error message.

awk '{a=$0; getline b; getline c; 

     if ( getline > 0 ) {print a, b, c, $0 } 

     else { print "Ohi " > "/dev/stderr" ; exit 65; }  }' data

where

• 
  a=$0; puts the full line in the variable a
  


• 
  getline b; reads a line and puts the variable b
  


• 
  getline c; obscure unfathomable command :-)
  


• 
  if (getline) if it is able to read a line...


• ..............{print a, b, c, $0} prints the 4 lines


• 
  else prints an error on the stderr device (screen or other) you can custom here...


• 
  exit 65 return an exit code different from 0 --->error
  

Bonus: why 65?

Searching for a good value for your exit code ^[2] you may found that it is suggested to see in /usr/include/sysexits.h among some C standards...

  #define EX_DATAERR      65      /* data format error */

65 is the most appropriate for the a data format error...

Honestly as answer I preferred 42,

but each value different from zero (and not reserved^[2]) could be good and 65 is the specific one...