Csdrhrt

I am trying to run a python program on a data processing (linux) cluster. The program runs fine if I ssh to the cluster from my linux desktop and start it but fails if I ssh to the cluster from my mac laptop and try to run it I get the following error:

[muzzle@cluster]~% acme-calibrate detector DARK --in-folder /gpfs/acme/exp/instrument/201802/experiment1/raw --out-folder /home/muzzle/dark_out --run 28

Traceback (most recent call last):

  File "/home/boff/calframework/framework/extern/bin/acme-calibrate", line 11, in <module>

    load_entry_point('acme-Offline-Calibration', 'console_scripts', 'acme-calibrate')()

  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 480, in load_entry_point

  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 2691, in load_entry_point

  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 2322, in load

  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 2328, in resolve

  File "/home/boff/pycalibrate_tmp/acme_calibrate/calibrate.py", line 236, in <module>

nb = nbformat.read(f, as_version=4)

  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/nbformat/__init__.py", line 141, in read

return reads(fp.read(), as_version, **kwargs)

  File "/home/boff/calframework/framework/extern/lib/python3.4/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 150: ordinal not in range(128)

It seems to be related to some conversion from UTF to ASCII, but I'm out of my depth.

I get the same error with both terminal.app and iterm2. Btw I used to have the iterm2 shell integration installed but I have removed them now.

The locale on the laptop is:

$ locale

LANG=

LC_COLLATE="C"

LC_CTYPE="UTF-8"

LC_MESSAGES="C"

LC_MONETARY="C"

LC_NUMERIC="C"

LC_TIME="C"

LC_ALL=

The locale on the remote server is:

% locale

locale: Cannot set LC_CTYPE to default locale: No such file or directory

locale: Cannot set LC_ALL to default locale: No such file or directory

LANG=en_US.UTF-8

LC_CTYPE=UTF-8

LC_NUMERIC="en_US.UTF-8"

LC_TIME="en_US.UTF-8"

LC_COLLATE="en_US.UTF-8"

LC_MONETARY="en_US.UTF-8"

LC_MESSAGES="en_US.UTF-8"

LC_PAPER="en_US.UTF-8"

LC_NAME="en_US.UTF-8"

LC_ADDRESS="en_US.UTF-8"

LC_TELEPHONE="en_US.UTF-8"

LC_MEASUREMENT="en_US.UTF-8"

LC_IDENTIFICATION="en_US.UTF-8"

LC_ALL=

This is the locale on my linux desktop, it is identical to that of the cluster:

$ locale

LANG=en_GB.UTF-8

LANGUAGE=en_GB:en

LC_CTYPE="en_GB.UTF-8"

LC_NUMERIC="en_GB.UTF-8"

LC_TIME="en_GB.UTF-8"

LC_COLLATE="en_GB.UTF-8"

LC_MONETARY="en_GB.UTF-8"

LC_MESSAGES="en_GB.UTF-8"

LC_PAPER="en_GB.UTF-8"

LC_NAME="en_GB.UTF-8"

LC_ADDRESS="en_GB.UTF-8"

LC_TELEPHONE="en_GB.UTF-8"

LC_MEASUREMENT="en_GB.UTF-8"

LC_IDENTIFICATION="en_GB.UTF-8"

LC_ALL=

It looks like the correct solution was to unset "Set locale environment variables on startup" in the terminal advanced options as described in https://blog.remibergsma.com/2012/07/10/setting-locales-correctly-on-mac-osx-terminal-application/

To do the same thing on iterm2 you have to unselect Preferences > Profiles > Terminal > Set locale variables automatically, as described here https://apple.stackexchange.com/questions/83428/lang-variable-wrong-in-iterm2-wheres-the-bug

The post
OS X Terminal UTF-8 issues
has some useful advice, based on
A guide to help set up Terminal Apps:

1. 
   

   In Terminal->Preferences->Settings->Advanced:

   
   
   
   
   

   • Under International, ensure the character encoding is set to Unicode (UTF-8).

   
   

   • Under Emulation, ensure that Escape non-ASCII input with Control-V is not set.

   
   
   
   

   This by itself may be enough to solve the problem.

   
   


2. 
   

   Use the command locale and look at the LC_CTYPE line, which should
   end with .UTF-8.
   If it doesn't, then in bash .profile or .bashrc in the home directory,
   add a line like this:

   
   
   

   export LC_CTYPE=your-locale-here.UTF-8
   
   

   
   


3. 
   

   Add the following lines to .inputrc in the home directory (create it if necessary):

   
   
   

   set meta-flag on
   
   set input-meta on
   
   set output-meta on
   
   set convert-meta off
   
   

   
   

You need to restart bash for the new settings to take effect.