Monday, July 20, 2020

Upgrade to MacOS 10.15.6 (Catalina), Some Bookkeeping Notes


Sportsfans,

After upgrading my Macs to MacOS X 10.15.6 Catalina last weekend, a number of compiling links broke and also some third-party software was "relocated".  Because I had delayed upgrading the MacOS from Mojave for over 9 months, the third-party software had adjusted to the new situation.  I downloaded a fresh version of "fink" (rev 0.45.1)   and installed it in the directory /opt/sw, because its old location /sw had been expunged by Apple's update.  This required some changes in my .cshrc file, so that I could point to the correct directories.  I had to upgrade XCode upward three versions, from 10.3 to 10.6, install the Xcode command tools, selfupdate fink, and then run an endless installation procedure to install my favorite X11 text editor nedit.    Took some time, but it all worked.

The older gcc5 compilers in fink were relocated by the MacOS upgrade and had been upgraded to gcc10 in fink.  However, I discovered that my fortran codes were compiling just fine, which meant that they were not using the gfortran compiler in fink.  I found the working gfortran compiler in Anaconda, /Users/jjpark/anaconda/bin/gfortran, so I dropped my .cshrc  pointers to the fink directories for gcc and let them utilize the Anaconda gcc compiler.  The X11 libraries also moved, into /opt, but changing the links fixed that.

So far, so good.  Crossed fingers.

Fearless Leader

Friday, June 5, 2020

Running R in Jupyter Notebooks within Anaconda

Running R in Jupyter Notebooks within Anaconda


### you can update packages in the GUI Anaconda Navigator, and create special environment to run R notebooks
#
### to run R in Jupyter notebooks, the Anaconda manual pages suggest that you create an R environment within Anaconda
### see https://docs.anaconda.com/anaconda/navigator/tutorials/r-lang/
### the recommended steps involve clicking the "Environments" tab in the left margin of the Navigator
### click "Create" at the bottom of the second Navigator column, under the list of existing environments
### give your new R environment a recognizable name, such as R-Land or r-tutorial
### click the python and R boxes to include both packages in the environment
### wait a bit until the Create button becomes live in the dialogue box.
### click "create" and go get a cup of coffee.
### When you wish to run or to write an R-based Jupyter notebook, click on this new environment,
### then follow the menu to "Open With Jupyter Notebook"

Monday, April 13, 2020

Running ObsPy in Jupyter Notebooks


Following the guide at

https://seismo.berkeley.edu/wiki_cider/images/1/10/Waite_cider_2019_tutorial.pdf


one sets up the ObsPy environment in Anaconda with the packages needed to make the notebooks go.

To review, setting up the ObsPy environment may have been done already, but if not, FIRST run these commands in a terminal window:

conda config --add channels conda-forge


• The above command sets conda to install packages from a certain place (conda-forge.org). The next command creates an environment called obspy (but you can call it just about anything) with python version 3.7.

conda create -n obspy python=3.7

• Next you will activate that newly created environment, in my installation I use "source" rather than "conda" for the activate command


source activate obspy

• You will notice that the command line prompt now has the name of the environment in parentheses (obspy). Now install obspy, and all the packages it depends on:


conda install obspy

conda will look for the obspy package at conda-forge, download it, and install it. You will be prompted a couple times for confirmation of download/install.

You have installed the obspy package, but there are a few more things that are useful to add. This one adds functionality to jupyter notebook to allow for environment switching

conda install nb_conda

• This should already be installed, but check.  In my case I had already installed basemap in my obspy environment

conda install basemap

• This adds more plotting functions.

conda install ipympl

• And, finally

 conda install jupyter


Using jupyter notebook

• Jupyter notebook is a web-browser tool that allows you to create python scripts and run then in an interactive mode. It a bit easier to use than the command line in python so lets move to that
  1. open terminal/xterm (osx or linux)
  2. activate the obspy environment on the command line
  3. then start jupyter notebook from the command line
  4. This will open a new browser window with the jupyter notebook
Now you can create a new notebook or an existing one (they have file extensions (.ipynb for ipython notebook)

cd Desktop/RData/ObsPy
source activate obspy
jupyter notebook

At this point you can cut and paste python commands into the notebook boxes and execute them one by one.  The notebook can be saved as an *.ipynb file for later use.  Be careful which directory you are in, or else you will be storing notebook files in your home directory.

Thursday, March 12, 2020

C-compilers that no longer work, because they cannot find stdio.h


In the gcc and cc compilers on my laptop, the stdio.h is sought in the directory /usr/include, which in traditional UNIX/LINUX/MacOS installations has contained a boatload of header files (*.h).  Apple has eliminated this directory.  After much searching on message boards, I found this directory on my laptop:

/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include

I copied the directory into the /usr directory

sudo cp -r /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include /usr

in order to write into a root-owned directory.  Once in place, and after refreshing the shell, I avoided the stdio.h error when compiling a sample C subroutine.  I get a number of other warnings, so Im not eager to write more C code in the future, but the command cc -c refft.c produces an object file refft.o that might work in your application.

Sunday, February 23, 2020

Installing ObsPy 1.1.1 in J.Park's MacOS 10.14.6 (Mojave) environment via Anaconda


Restarting ObsPy after a hiatus requires that one updates all the packages.  There are many websites out there to advise you, but I didn't find one whose advice did the job, exactly.  So I will report my experience updating to Obspy 1.1.1.

I had Anaconda installed, so the best direction I found was within the Github web pages

https://github.com/obspy/obspy/wiki/Installation-via-Anaconda

but my upgrade/installation deviated from their recipe.  I had to update my version of conda, the program that installs packages within Anaconda.  Follow the web pages on that.  From a terminal window. follow the advice from the GitHub pages.  Add the condo-forge channel to enable downloading software packages.  From the pages, I cut and pasted thusly:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ conda config --add channels conda-forge
  • We strongly recommend to work with separate Anaconda environments, and especially not use the special base environment (formerly named root on conda versions <4 .4="" all="" code="" environment="" for="" is="" nbsp="" style="background-color: rgba(27, 31, 35, 0.0470588); border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.600000381469727px; margin: 0px; padding: 0.2em 0.4em;" this="" used="">conda
 commands and environment manipulations, package installations etc.) for anything besides updating the conda package itself (if necessary):
$ conda create -n obspy python=3.7   # or e.g. python=2.7
$ conda activate obspy  # this command used to be 'source activate obspy' on older conda versions < 4.4
(obspy) $ 
  • Install pre-compiled ObsPy conda package from Anaconda cloud with:
(obspy) $ conda install obspy
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Two caveats on this recipe.  First, the ObsPy documentation suggests that python=3.7 is not supported by ObsPy versions up to 1.1.1, so I replaced "3.7" with "3.6" for the python version.  Next, the command "conda activate obspy" does not work for me.  I tried the recommended command "conda init bash" to make these commands work, but my computer protested a mis-application of the init-command.  Instead I used the older command for activating the obspy channel: "source activate obspy"  After the obspy channel was activated, I ran "conda install obspy" successfully.
NOTE:  I had installed an earlier version of obspy in Fall 2017, but I was unable to update it with simple commands like "conda update obspy".  Perhaps there were too many packages to update cleanly after 2+ years of neglect.  So I removed the obspy package and did a fresh install "conda install obspy".  Getting the sequence of commands to enact the procedure of updating my installation, then removing it and starting fresh (though without re-installing Anaconda!) took me several days of trouble-shooting.  I replicated the procedure, without the trial-and-error detours, on a desktop iMac.  The procedure was first to update conda, then delete the obspy package, then activate the obspy channel for working with obspy.  Among other features, the obspy channel deprecates the version of python to v3.6.7 to avoid conflicts with python v3.7. Then install the obspy package from scratch.  

Friday, June 1, 2018

Preparing IRIS data files for analysis by R, Take 1

Preparing IRIS data files for analysis by R: converting 2-column ascii to CSV files.  Rdseed and the IRIS DMS produce datafiles in  2-column ASCII format with *.ascii filename suffixes.  These files start with a one=-line header, then a UTC time and a datum in the form of instrument counts

TIMESERIES CU_BCIP_00_LHZ_M, 180001 samples,  1.0 sps, 2016-11-13T10:00:00.000000, TSPAIR, FLOAT, Counts
2016-11-13T10:00:00.000000             3130
2016-11-13T10:00:01.000000             1921


This command filters all the ascii files for a minimum size, to avoid data-window fragments

ls -l *ascii | awk ' $5>NNNN {print $9}' >! infiles

Prepare a shell command file for using awk to strip out the data column and write a CSV file.  First construct the CSV filenames from the key attributes in the IRIS filename: Station.Location.Channel.csv

# construct CSV filenames from long IRIS filenames

sed 's/\./ /g' infiles >! blob
awk '{print  $1 "."  $2 "."  $3 "."  $4 "."  $5 "."  $6 "."  $7 "."  $8 "."  $9 "."  $10 "."  $11 " "  $7 "."  $8 "."  $9 ".csv" }' blob  >! infiles1 

-> head infiles1
2016.318.10.00.00.0000_CU.ANWB.00.LHZ.M.ascii ANWB.00.LHZ.csv
2016.318.10.00.00.0000_CU.BBGH.00.LHZ.M.ascii BBGH.00.LHZ.csv
2016.318.10.00.00.0000_CU.BCIP.00.LHZ.M.ascii BCIP.00.LHZ.csv
2016.318.10.00.00.0000_CU.GRGR.00.LHZ.M.ascii GRGR.00.LHZ.csv
2016.318.10.00.00.0000_CU.GRTK.00.LHZ.M.ascii GRTK.00.LHZ.csv
2016.318.10.00.00.0000_CU.GTBY.00.LHZ.M.ascii GTBY.00.LHZ.csv

Write the shell script that runs awk once per filename, for LHZ files. 

# because I cant figure out how to add a quote to the sed output, the script 
# for generating CSV data files from LHZ ascii data requires a file edit
awk ' {print "awk NR>1 {print NR-1 \", \" $2} " $1 " >! " $2 "; wc "$2 } ' infiles1 >! blob
te blob &
#save as saveblob
source saveblob

note that the NR>1 skips the header line.  The time ordinate is indexed to the record number

-> head blob
awk NR>1 {print NR-1 ,  $2} 2016.318.10.00.00.0000_CU.ANWB.00.LHZ.M.ascii >! ANWB.00.LHZ.csv; wc ANWB.00.LHZ.csv
awk NR>1 {print NR-1 ,  $2} 2016.318.10.00.00.0000_CU.BBGH.00.LHZ.M.ascii >! BBGH.00.LHZ.csv; wc BBGH.00.LHZ.csv
awk NR>1 {print NR-1 ,  $2} 2016.318.10.00.00.0000_CU.BCIP.00.LHZ.M.ascii >! BCIP.00.LHZ.csv; wc BCIP.00.LHZ.csv

note that we edited blob --> saveblob by adding some quote marks around the awk command. 

-> head saveblob
awk 'NR>1 {print NR-1", " $2}' 2016.318.10.00.00.0000_CU.ANWB.00.LHZ.M.ascii >! ANWB.00.LHZ.csv; wc ANWB.00.LHZ.csv
awk 'NR>1 {print NR-1", " $2}' 2016.318.10.00.00.0000_CU.BBGH.00.LHZ.M.ascii >! BBGH.00.LHZ.csv; wc BBGH.00.LHZ.csv
awk 'NR>1 {print NR-1", " $2}' 2016.318.10.00.00.0000_CU.BCIP.00.LHZ.M.ascii >! BCIP.00.LHZ.csv; wc BCIP.00.LHZ.csv

Write the shell script that runs awk once per filename, for VHZ files. Note that the time ordinate is converted to ksec from seconds.  Time-step increments of 0.01 ksec = 10 seconds.

ls -l *ascii | awk ' $5>3700000 {print $9}' >! infiles
# because I cant figure out how to add a quote to the sed output, the script 
# for generating CSV data files from VHZ ascii data requires a file edit
awk ' {print "awk NR>1 {print (NR-1)/100. \", \" $2} " $1 " >! " $2 "; wc "$2 } ' infiles1 >! blob
te blob &
#save as saveblob
source saveblob

Tuesday, September 5, 2017

Machine Learning Python: Wavelet Spectrum examples


Another test on my MacBook Pro:  The wavelet transform example in Megies et al (2011).

You may need to install the Machine Learning Python package.  The command

conda install mlpy

did not work for me, so I search the internet for a method to install mlpy

The mlpy package requires the GNU Scientific Library (GSL), which was not installed on my laptop, and may not be installed on yours either.  This package can be found on the web.  My method was taken from the book "Practical Data Analysis Cookbook" by Tomasz Drabas.  First, direct your web browser to

http://gnu.mirro.iweb.com/gsl

and download the latest version of GSL, typically called gsl-latest.tar-gz. Currently the latest is V2.4. Move the downloaded file into your Applications directory (on a Mac) -- I dont know the equivalent place on a Windows machine.  Then unpack this gzipped tarfile by double-clicking on it ot else use the command

gzip -c gsl-latest.tar-gz | tar -xv

The unpacked file will produce a folder with the GSL software in it.  In your terminal-app go to this folder and prepare and install GSL

./configure 

make

make install

Hopefully this compilation and installation proceeds without error.  Now download a compressed tarfile with the lastest version of mlpy from this website into your anaconda dirctory, which on my laptop is /Users/jjpark/anaconda.  The current latest version is V3.5.

http://sourceforge.net/projects/mlpy/files

Double-click the tarfile to extract its contents into a subdirectory, or else

gzip -c mlpy-.tar.gz | tar -xv

or, if already uncompressed,

tar -xvf mlpy-.tar

This action produces a subdirectory mlpy-.  cd to this directory in your terminal app and type

python setup.py install

and mlpy will be installed if all goes well.  Python modules that one obtains from sourceforge and other websites typically include a setup.py scriptfile to instruct python how to set the module up for later use.

I discovered coding bugs in the mlpy module while trying to execute the coding examples.  The continuous wavelet transform relies on a script file ./mlpy-3.5.0/mlpy/wavelet/continuous.py that had a mistake in its Line 162:

161     J = floor(dj**-1 * log2((N * dt) / s0))
162     s = empty(J + 1)
   
The python function "floor" from the numpy module returns a floating-point number, but the python function "empty" expects an integer to define how long an array to generate. This combination of mismatched variable types leads to an error message:

'numpy.float64' object cannot be interpreted as an integer

Fixing the code to read, e.g.,

     J = floor(dj**-1 * log2((N * dt) / s0))
     JJ = int(J)
     s = empty(JJ + 1)

*should* solve the problem, but it doesn't.  Why? Because the original continuous.py is loaded into the python libraries during the install-operation, and changing the code after installation doesnt change the installed code.  You must make changes to the script continuous.py in the mlpy-3.5.0 subdirectory, and then re-install the module by executing

python setup.py install

while the terminal app is "in" the ./mlpy-3.5.0 directory.  Note that python will tell you that the execution error resides in a copy of continuous.py that lies deep within a subdirectory tree.  This is NOT the copy that needs to be debugged.  You must debug the copy of continuous.py that sits close within the subdirectory anaconda/mlpy-3.5.0/mlpy/wavelet, and then re-install the mlpy module.

Here is a simple example of a wavelet transform of a randomly-generated array of numbers

import numpy as np
import matplotlib.pyplot as plt
import mlpy.wavelet as wave
x = np.random.sample(512)
scales = wave.autoscales(N=x.shape[0], dt=1, dj=0.025, wf='morlet', p=2)
X = wave.cwt(x=x, dt=1, scales=scales, wf='dog', p=2)
fig = plt.figure(1)
ax1 = plt.subplot(2,1,1)
p1 = ax1.plot(x)
ax1.autoscale_view(tight=True)
ax2 = plt.subplot(2,1,2)
p2 = ax2.imshow(np.abs(X), interpolation='nearest')
plt.show()


Here is the wavelet transform of the data series we downloaded in the first ObsPy exercise

import numpy as np, matplotlib.pyplot as plt
import mlpy 
import mlpy.wavelet as wave
from obspy.core import UTCDateTime
from obspy.clients.arclink import Client
from obspy.signal.filter import lowpass
from obspy.signal.invsim import corn_freq_2_paz, simulate_seismometer
client = Client("webdc.eu", port=18001)
t = UTCDateTime("2009-08-24 00:20:03")
st = client.get_waveforms("BW", "RJOB", "", "EHZ", t, t+30)
omega0 = 2.0
trace = st[0].data
npts = len(trace)
dt = st[0].stats.delta
dj = 0.05
wf = 'morlet'
p=omega0
scl = mlpy.wavelet.autoscales(npts, dt, dj , wf, p) 
spec = mlpy.wavelet.cwt(trace, dt, scl , wf='morlet', p=omega0) 
freq = (omega0 + np.sqrt(2.0 + omega0**2)) / (4*np.pi * scl[1:])
t = np.arange(st[0].stats.npts) / st[0].stats.sampling_rate 
plt.imshow(np.abs(spec), extent=(t[0], t[-1], freq[-1], freq[0])) 
plt.xlabel('Time [s]') 
plt.ylabel('Frequency [Hz]') 
plt.show()

Try these scripts to be sure that your installation of Python and ObsPy is working properly.








 
Link