Monday, December 13, 2010

Converting GILDAS-CLASS data cubes (lmv files) to fits

As usual, CLASS documentation is nearly impossible to navigate. At the end of the CLASS "introduction" (gildas-intro.pdf) there is a subtle and obscure reference to the vector\fits command. The conversion is actually relatively straightforward:

vector\fits outfile.fits from infile.lmv



AG

Saturday, November 27, 2010

IDL syntax highlighting in VIM

I've edited my idlang.vim to auto-identify files that start with a semicolon.

Add Line 19:
syn match idlangStatement "^\s*;\s"

Line 61/2 (allow spaces before ;):
syn match idlangComment "\s*[\;].*$" contains=idlangTodo


AG

Monday, November 22, 2010

Mercurial - behave like SVN?

I'm trying to use hooks to make mercurial behave like svn when committing. I like the idea that I can commit changes to my cloned repo while I'm away from the internet, but I never want that behavior when I do have internet access. Therefore, I want to attempt to pull before updating and attempt to push after committing. Every time. I have been consistently very unhappy with the hg merge command.


[hooks]
precommit = hg pull; hg up
postcommit= hg push
post-pull = hg up


However, this doesn't work. precommit freezes with the error
waiting for lock on working directory of [dir] held by [procnum]

and pre-commit results in other errors:

running hook pre-commit: hg pull; hg up
pulling from [source]
searching for changes
no changes found
running hook post-pull: hg up
abort: outstanding uncommitted merges
warning: post-pull hook exited with status 255
abort: outstanding uncommitted merges
warning: pre-commit hook exited with status 255


AG

Wednesday, November 3, 2010

Repositories for observers

I should have posted these a while ago....
casaradio is a subversion repository for folks at The Center for Astrophysics and Space Astronomy at CU Boulder to post radio astronomy related codes. So far, emphasizes single dish (GBT, Arecibo), but will include EVLA, CARMA, and ALMA eventually.
aposoftware is a similar page, but is a mercurial repository and is meant to include instrument-specific software for the Apache Point Observatory 3.5m telescope. Right now includes a TUI script or two and the TSPEC and DIS IRAF-twodspec pipelines.

I'd be remiss to leave out the BGPS pipeline even though it's mentioned on the previous post.

Also, agpy is my personal code repository.

BGPS data paper published

Metalinking! The BGPS paper finally made it onto astro-ph today. It will be published in ApJS before the year's end.

Links to all of the published BGPS papers at the Bolocam Data Team website

And just because I want more linking, here they all are again:
The Bolocam Galactic Plane Survey I. Survey Description and Data Reduction arXiv
The Bolocam Galactic Plane Survey II. Catalog of the Image Data arXiv
The Bolocam Galactic Plane Survey III. Characterizing Physical Properties of Massive Star-Forming Regions in the Gemini OB1 Molecular Cloud arXiv
The Bolocam Galactic Plane Survey IV: λ = 1.1 and 0.35 mm Dust Continuum Emission in the Galactic Center Region

The same set of links is reproduced at the pipeline googlecode page.

Tuesday, October 5, 2010

Next Generation Astronomy

A brief commentary on a recent article Next Generation Astronomy :

I'm probably one of the folks who will be working on making that future happen, in the sense of being the intermediary between the end-user and the instrumentalist. Scientists are still needed to build instruments, but those who build the instruments don't necessarily have the time or expertise to build the software pipelines that generate high-quality data.

At the same time, the STARLINK folks do a much better job than I do, and I already spend about half of my time comparing different archival data sets... so I think his crystal ball is overall very accurate.

Tuesday, September 14, 2010

Converting a CLASS-created .fits file to a real (FITS-compliant) FITS file

This post is to remind me, the next time I go looking, how the hell to convert from a GILDAS CLASS fits spectrum (created by fits write blah.fits /mode spectrum) to a FITS-compliant spectrum.

First, remember the FITS-WCS spectral definitions: http://www.aanda.org/index.php?option=com_article&access=bibcode&Itemid=129&bibcode=2006A%2526A...446..747GFUL


And the peculiar CLASS definitions: http://iram.fr/IRAMFR/GILDAS/doc/html/class-html/node84.html

Key points:
CLASS stores the CDELT parameter as DELTAV in m/s instead of km/s and the velocity offset of the spectral frame in VELO-LSR also in m/s.

Things to set:
CTYPE = VRAD
SPECSYS = SOURCE
SSYSSRC = LSRK
VELOSYS = frame velocity (VELO-LSR or CRVAL1)

This information is subject to change...

Wednesday, August 18, 2010

Observing @ GBT: Signs of good & bad data

So far, all of the observations for the H2CO densitometry project have been performed at the Green Bank Telescope. During a 10-day long observing trip here, I've learned a lot about diagnosing bad data.




This first image shows TSYS vs Airmass for good data. The high outliers are just sources with continuum in them - the continuum is the source of the extra signal, not atmosphere.  The receiver temperature is a nice 20.6 K, and you get about 5 K extra per airmass, suggesting a zenith optical depth of 0.018 assuming a round atmospheric temperature of 300K.



In the same style plot, there is a set of observations with low system temperatures: that stuff is good. There is also a set with clearly rising system temperatures, even at constant elevation. These data are bad. During this observation, the "blowers" that are meant to keep dew off of the receivers failed. Dew buildup on the receiver covers lead to higher optical depths and therefore system temperatures.




Finally, this data set was totally useless. Ku-band is not particularly sensitive to water in the atmosphere... but it's still not a good idea to observe during a rain storm.  Note that the fitted receiver temperature TREC is nonsensical.

Tuesday, August 17, 2010

Galactic Formaldehyde Densitometry Project

The Galactic Formaldehyde Densitometry Project is finally official and on its way. I am PI of a joint Arecibo / Green Bank Telescope project to survey formaldehyde, H2CO, in a sample of ~400 "dust clumps" in the Galactic Plane. The clumps represent condensations within larger molecular clouds that are likely to be forming stars, particularly massive stars and star clusters.

Formaldehyde is uniquely useful for measuring densities of these gas clumps. Because of a collisional selection effect, the low-lying K-doublet transitions of H2CO undergo "anti-inversion", which means their lower-energy states are overpopulated relative to the higher states, and it can therefore be seen in absorption against the cosmic microwave background. This unique property removes a key bias present in emission line studies in which variable excitation conditions can dominate the observability of a line.

For the next few posts, I will include calibration and data reduction information from the observations in this survey. The Green Bank observations are well underway, with nearly half of the sources observed (though many at poor sensitivity) and the Arecibo observations have been approved.

Monday, August 16, 2010

Neat new things....

1. sptool is a quick way to compare standards to stellar spectra. Nice, I'd been looking for a tool like that.
2. GNU screen captions are useful especially when working in a screen-within-a-screen environment (who does that, really?)
3. finally got SPLAT to work... turns out I just hadn't reduced my damned data
4. kill -STOP and kill -CONT are really useful ways to pause programs that are sucking up resources if you want to resume them later. Haven't tried this on "real" code yet.

Friday, August 13, 2010

Filled step plots in matplotlib

It's not possible to do a simple filled step plot in matplotlib using default
commands. Workaround:


def steppify(arr,isX=False,interval=0):
"""
Converts an array to double-length for step plotting
"""
if isX and interval==0:
interval = abs(arr[1]-arr[0]) / 2.0
newarr = array(zip(arr-interval,arr+interval)).ravel()
return newarr

plot(xx,yy,linestyle='steps-mid',color='b',linewidth=1.5)
fill_between(steppify(xx[x1:x2],isX=True),
steppify(yy[x1:x2])*0,
steppify(yy[x1:x2]),
facecolor='b',alpha=0.2)

Wednesday, July 28, 2010

Histogram in Google Spreadsheet

It's not easy to make a histogram in google spreadsheets without replicating data. The "countif" function would be great, except it only allows very simple criteria. However, there's a workaround:
=count(Filter('Grades'!V2:V30,'Grades'!V2:V30>0.9))
=count(Filter('Grades'!V2:V30,'Grades'!V2:V30<0.9,'Grades'!V2:V30>0.8))

The Filter() function returns an array, which can be operated on like any other set of cells.

It's still not easy to make a nice-looking histogram, but the output of this process is at least usable.

Friday, July 9, 2010

IDL-to-Python

astrobetter started up an idl-to-python guide on their wiki.

Sunday, June 13, 2010

Ghostscript error?

I've been receiving the following error when attempting to compile (ps2pdf) my w5 outflows paper:

Error: /rangecheck in --get--
Operand stack:
pdfmark --dict:20/25(ro)(L)-- --nostringval-- 50
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1878 1 3 %oparray_pop 1877 1 3 %oparray_pop 1861 1 3 %oparray_pop 1755 1 3 %oparray_pop --nostringval-- %errorexec_pop .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- %finish_show --nostringval-- --nostringval-- 8 6 1 --nostringval-- (pdf_text_enum_t) %op_show_continue --nostringval--
Dictionary stack:
--dict:1153/1684(ro)(G)-- --dict:0/20(G)-- --dict:71/200(L)-- --dict:125/300(L)-- --dict:44/200(L)-- --dict:138/224(L)--
Current allocation mode is local
Last OS error: 2
Current file position is 267478928
GPL Ghostscript 8.71: Unrecoverable error, exit code 1

I get the same error with Ghostscript 8.64, but on my laptop, using the fink version, it works. Similarly, there are errors with the postscript, so I'm led to believe it's an error in latex:

$ latex --version
pdfTeX 3.1415926-1.40.10-2.2 (TeX Live 2009)
kpathsea version 5.0.0
Copyright 2009 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
There is NO warranty. Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
Compiled with libpng 1.2.39; using libpng 1.2.39
Compiled with zlib 1.2.3; using zlib 1.2.3
Compiled with xpdf version 3.02pl3

No idea what the cause is but it's time to start documenting steps and looking for a workaround. Compiling on the lappy isn't a good option.

Wednesday, May 26, 2010

EVLA information

It has been really hard to find EVLA information like beam size, largest angular scale, sensitivity, etc. on the VLA pages because all of the google searches point to old VLA information. The most useful and recent EVLA information on beam size and largest angular scale is here

Tuesday, May 25, 2010

usetex failure in latex documents

When I use matplotlib's internal tex (rcParams['text.useTex']=False), the postscript files generated cause errors that look like this when you try to ps2pdf them:


ps2pdf h2co_pilot.ps
Error: /rangecheck in --get--
Operand stack:
--dict:20/25(ro)(L)-- --nostringval-- 71
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1862 1 3 %oparray_pop 1861 1 3 %oparray_pop 1845 1 3 %oparray_pop 1739 1 3 %oparray_pop --nostringval-- %errorexec_pop .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- %finish_show --nostringval-- --nostringval-- 9 6 0 --nostringval-- (pdf_text_enum_t) %op_show_continue --nostringval--
Dictionary stack:
--dict:1147/1684(ro)(G)-- --dict:0/20(G)-- --dict:70/200(L)-- --dict:116/300(L)-- --dict:44/200(L)-- --dict:25/42(L)--
Current allocation mode is local
Last OS error: 2
Current file position is 791626
GPL Ghostscript 8.64: Unrecoverable error, exit code 1
make: *** [h2co_pilot.pdf] Error 1


They will not open in MacOS's Preview.app either.

Solution: Make figures with rcParams['text.useTex'] = True

Saturday, May 1, 2010

RATRAN on Mac OS X

Mac OS X doesn't like the defaults built in to RATRAN. It died unhappily with errors like:

ld_classic: can't locate file for: -lcrt0.o
and
ld: warning: in /usr/local/lib//libcfitsio.a, file is not of required architecture


In order to get it to run, I had to do the following:
  1. Install CFITSIO with CFLAGS="-arch x86_64 -arch i386 -g -O2" to /usr/local/lib
  2. Edit the sky/Makefile OPT variable (line 23) to read:
    OPT = -I. -O2 -fno-automatic -arch x86_64
Also, you need to set up system variables:

export RATRAN=/path/to/Ratran
export RATRANRUN=/path/to/Ratran/run

Thursday, April 22, 2010

Gildas CLASS

It's absurdly difficult to find help on GILDAS Class, probably because you can't google "class" and most people probably don't label every piece of code with "GILDAS class".

Anyway, here are some scripts that I refer back to often:

file in August2009BGPS.dat
file out August2009fits.dat multiple
on error "file out August2009fits.dat"
say "READ IN FILES"
define character sourcelist*10[300]
accept sourcelist /column observed_sources.txt
on error "continue"
get 1001
set window -100 160
set mask -400 -100 160 400
set mode x -400 400
set align velocity
for i 1 to 161
say "Working on SOURCE "'i'
find /source 'sourcelist[i]' /telescope "CSO 4GHZ IF1" /offset 0 0 /quality 5
average
on error "@avplot2 'sourcelist[i]' 'i'; next"
base 3
line 0
min
plot
vis
write i
! on error "continue"
next


and


file in araya-2004.cls
find
define character filename*20
for i 1 to 20
say "Working on source "'i'
get next
let filename "araya-2004_"'i'".fits"
say "fits write "'filename'" /mode spectrum"
fits write 'filename' /mode spectrum
next

!file in araya-2002.cls
!find
!define character filename*20
!for i 1 to 42
! say "Working on source "'i'
! get i
! let filename "araya-2002_"'i'".fits"
! say "fits write "'filename'" /mode spectrum"
! fits write 'filename' /mode spectrum
!next

Thursday, April 15, 2010

Montage wrapper

(I'm going to try to gradually shift my blogging to this one...)

I wrote a bash wrapper for Tom Robitaille's montage wrapper to allow fits wildcards.


#!/bin/bash

origdir=`pwd`

#echo $# $*

if [ $# -gt 0 ]
then
for ii in $*
do
if [ ${ii%=*} == 'header' ]
then
/usr/local/bin/montage/mGetHdr ${ii#*=} mosaic.hdr
elif [ ${ii%=*} == 'outfile' ]
then
outfile=${ii#*=}
elif [ `echo $ii | grep =` ]
then
params="$params,${ii%=*}='${ii#*=}'"
elif [ `echo $ii | grep ".fits"` ]
then
files=( ${files[@]} $ii )
fi
done
fi
echo ${files[@]} ${#files}
if [ ${#files} -gt 0 ]
then
mkdir tmp
cp ${files[@]} tmp/
cp mosaic.hdr tmp/
cd tmp/
fi

if [ -f mosaic.hdr ]
then
echo "mosaic.hdr exists, continuing"

dir=`pwd`
echo python -c "import montage; montage.wrappers.mosaic('$dir','$dir/mosaic',header='$dir/mosaic.hdr'$params)"
python -c "import montage; montage.wrappers.mosaic('$dir','$dir/mosaic',header='$dir/mosaic.hdr'$params)"

cd $origdir
if [ -d tmp ]
then
if [ $outfile ]
then
mv tmp/mosaic/mosaic.fits $outfile
else
mv tmp/mosaic mosaic
fi
rm -r tmp
fi

else
echo "mosaic.hdr does not exist. Quitting."
cd $origdir
fi

Sunday, March 28, 2010

IRAF append/prepend/replace

IRAF has special syntax to append/replace, so you can do something like:
imarith *.fits - dark.fits ds_//*.fits
...though I can't remember right now how to do replacement etc.

Thursday, March 18, 2010

command line & escape key

Problem: in both the bash command line and ipython, if I hit escape twice, I can never again do history-search-backwards (i.e. when you type part of a command and type "up" and reverse-search through your command history). Any idea how to fix this?

Friday, March 5, 2010

Rebuilding python from scratch again

I got scipy working a week or two ago, but doing so killed matplotib's tkagg. So, I switched to the MacOSX backend, which worked ok until I realized that the interactive (connect) features of macosx failed miserably. This led me to try to get matplotlib working.... which broke with those awful "symbol not found" errors in ft2font.so and _path.so, which I've determined all have to do with linking to the wrong library files.

The most worrisome part of this process was discovering that a full Time Machine recovery of /usr and /Library/Frameworks and /Library/Python did *not* restore python - it stayed dead with IDENTICAL errors. So there are probably additional layers of hidden links.

The process below is based on hyperjeff's blog post but differs substantially based on Sam Skillman's recommendations and the very big issue I ran into that my /usr/local files appeared to be corrupted. After this install, my path no longer includes /usr/local/bin and /sw has been moved to /_sw... hopefully one of these days I'll be ballsy enough to delete it.

  1. Install python 2.6.4
    1. Needed a clean terminal with no flags set at all. Don't know why - all I had set were a bunch of -arch x86_64 flags.

      export LD_LIBRARY_PATH="/usr/local/lib:/usr/X11/lib"
      ./configure --enable-framework=/Library/Frameworks MACOSX_DEPLOYMENT_TARGET=10.6
      make -j 17
      sudo make install

    2. Reset PYTHONPATH to blank
    3. alias clearflags='export CFLAGS=""; export CCFLAGS=""; export CXXFLAGS=""; export LDFLAGS=""; export FFLAGS="";' to make sure

  2. Install FFTW

    cd ~/tmp
    curl -O http://www.fftw.org/fftw-3.2.2.tar.gz
    tar xf fftw-3.2.2.tar.gz
    cd fftw-3.2.2
    clearflags
    ./configure CC="gcc -arch x86_64" CXX="g++ -arch x86_64" CPP="gcc -E" CXXCPP="g++ -E"
    make -j 17
    sudo make install

  3. Install UMFPACK

    cd ~/tmp
    curl -O http://www.cise.ufl.edu/research/sparse/umfpack/current/UMFPACK.tar.gz
    curl -O http://www.cise.ufl.edu/research/sparse/UFconfig/current/UFconfig.tar.gz
    curl -O http://www.cise.ufl.edu/research/sparse/amd/current/AMD.tar.gz
    tar xf AMD.tar.gz
    tar xf UFconfig.tar.gz
    tar xf UMFPACK.tar.gz
    sed -ibck 's/F77 = f77/F77 = gfortran/' UFconfig/UFconfig.mk
    sed -ibck '299,303s/# //' UFconfig/UFconfig.mk
    cp UFconfig/UFconfig.h AMD/Include/
    cp UFconfig/UFconfig.h UMFPACK/Include/
    cd UMFPACK
    make -j 17
    make hb
    make clean

  4. Install numpy
    1. Set environment variables

      export MACOSX_DEPLOYMENT_TARGET=10.6
      export CFLAGS="-arch x86_64"
      export FFLAGS="-m64"
      export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64"
      export PYTHONPATH="/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/"
      echo "[amd]
      library_dirs = /Users/adam/tmp/AMD/Lib
      include_dirs = /Users/adam/tmp/AMD/Include
      amd_libs = amd

      [umfpack]
      library_dirs = /Users/adam/tmp/UMFPACK/Lib
      include_dirs = /Users/adam/tmp/UMFPACK/Include
      umfpack_libs = umfpack" > site.cfg

    2. Setup & Install

      python setup.py build --fcompiler=gnu95
      sudo python setup.py install

    3. Test: python -c "import numpy"

  5. Install scipy. The important thing is to use g++-4.2 because g++-4.5 doesn't accept the -arch flag. Also, get rid of /sw if it's on your computer at all.

    sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
    cd ~/repos/scipy-0.7.1
    python setup.py clean
    rm -rf build
    clearflags
    FFLAGS="-m64" CFLAGS="-arch x86_64 -I/usr/local/include/freetype2 -I/usr/X11/include -L/usr/X11/lib" LDFLAGS="-Wall -undefined dynamic_lookup -bundle -lpng -arch x86_64" CXX="/usr/bin/g++-4.2" CC="/usr/bin/gcc-4.2" python setup.py build
    python setup.py install

    Test the install:

    python -c "import scipy, scipy.fftpack, scipy.interpolate"

    1. Install matplotlib. MAKE SURE /usr/bin/texbin is in front of /usr/local/bin and /sw/bin so that dvipng comes from MacTEX. I also ended up having to remove /usr/local/bin from my path completely

      sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
      cd ~/repos/matplotlib-svn
      python setup.py clean
      rm -rf build
      clearflags

    2. Do hyperjeff's recommended edits except don't use /usr/local because it's f'd:
      make.osx:

      MACOSX_DEPLOYMENT_TARGET=10.6
      PREFIX=/usr
      PYTHON=/Library/Frameworks/Python.framework/Versions/Current/bin/python

      ## You shouldn't need to configure past this point (and yet…)

      PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig"
      CFLAGS="-arch i386 -arch x86_64 -I${PREFIX}/include -I${PREFIX}/include/freetype2 -isysroot /Developer/SDKs/MacOSX10.6.sdk"
      LDFLAGS="-arch i386 -arch x86_64 -L${PREFIX}/lib -syslibroot,/Developer/SDKs/MacOSX10.6.sdk"
      FFLAGS="-arch i386 -arch x86_64"

      setup.cfg:

      wxagg = False

    3. Do the install (different from hyperjeff b/c I don't want root)
      sudo make -f make.osx fetch deps
      make -f mpl_build mpl_install
      python setup.py install


  6. Install setuptools
  7. easy_install ipython
  8. install everything else pythonically

Monday, March 1, 2010

Thursday, February 25, 2010

macvim crash

well, it finally happened.... my reliable, trusty editor crashed. That should be impossible. I am ready to call it quits for the week....


Process: MacVim [650]
Path: /Applications/Vim.app/Contents/MacOS/MacVim
Identifier: org.vim.MacVim
Version: 7.2 (49)
Code Type: X86 (Native)
Parent Process: Vim [649]

Date/Time: 2010-02-25 13:12:43.001 -0700
OS Version: Mac OS X 10.6.2 (10C540)
Report Version: 6

Interval Since Last Report: 871676 sec
Crashes Since Last Report: 26
Per-App Interval Since Last Report: 938504 sec
Per-App Crashes Since Last Report: 1
Anonymous UUID: 03159B9E-2257-4E38-8C4A-4D4DAF5641A7

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: 0x000000000000000d, 0x0000000000000000
Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Thread 0 Crashed: Dispatch queue: com.apple.main-thread
0 com.apple.CoreFoundation 0x99119480 __CFSetCallback + 0
1 com.apple.CoreFoundation 0x990c78bc ___CFBasicHashFindBucket1 + 444
2 com.apple.CoreFoundation 0x990cfaac CFBasicHashFindBucket + 252
3 com.apple.CoreFoundation 0x990e8293 CFSetGetValue + 131
4 com.apple.AppKit 0x961bae7e -[NSWindow _discardTrackingRect:] + 59
5 com.apple.AppKit 0x961badca -[NSView(NSInternal) _uninstallTrackingArea:] + 123
6 com.apple.AppKit 0x960d2c32 -[NSView(NSInternal) _uninstallRemovedTrackingAreas] + 293
7 com.apple.AppKit 0x960dac40 -[NSView(NSInternal) _updateTrackingAreas] + 646
8 com.apple.CoreFoundation 0x990ea4e0 CFArrayApplyFunction + 224
9 com.apple.AppKit 0x960daefb -[NSView(NSInternal) _updateTrackingAreas] + 1345
10 com.apple.CoreFoundation 0x990ea4e0 CFArrayApplyFunction + 224
11 com.apple.AppKit 0x960daefb -[NSView(NSInternal) _updateTrackingAreas] + 1345
12 com.apple.CoreFoundation 0x990ea4e0 CFArrayApplyFunction + 224
13 com.apple.AppKit 0x960daefb -[NSView(NSInternal) _updateTrackingAreas] + 1345
14 com.apple.AppKit 0x960da8db _handleInvalidCursorRectsNote + 392
15 com.apple.CoreFoundation 0x99135892 __CFRunLoopDoObservers + 1186
16 com.apple.CoreFoundation 0x990f218d __CFRunLoopRun + 557
17 com.apple.CoreFoundation 0x990f1864 CFRunLoopRunSpecific + 452
18 com.apple.CoreFoundation 0x990f1691 CFRunLoopRunInMode + 97
19 com.apple.HIToolbox 0x936f6f0c RunCurrentEventLoopInMode + 392
20 com.apple.HIToolbox 0x936f6bff ReceiveNextEventCommon + 158
21 com.apple.HIToolbox 0x936f6b48 BlockUntilNextEventMatchingListInMode + 81
22 com.apple.AppKit 0x960b0ac5 _DPSNextEvent + 847
23 com.apple.AppKit 0x960b0306 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 156
24 com.apple.AppKit 0x9607249f -[NSApplication run] + 821
25 com.apple.AppKit 0x9606a535 NSApplicationMain + 574
26 org.vim.MacVim 0x0000238b _start + 209
27 org.vim.MacVim 0x000022b9 start + 41

Thread 1: Dispatch queue: com.apple.libdispatch-manager
0 libSystem.B.dylib 0x98d0c0ea kevent + 10
1 libSystem.B.dylib 0x98d0c804 _dispatch_mgr_invoke + 215
2 libSystem.B.dylib 0x98d0bcc3 _dispatch_queue_invoke + 163
3 libSystem.B.dylib 0x98d0ba68 _dispatch_worker_thread2 + 234
4 libSystem.B.dylib 0x98d0b4f1 _pthread_wqthread + 390
5 libSystem.B.dylib 0x98d0b336 start_wqthread + 30

Thread 2:
0 libSystem.B.dylib 0x98ce58da mach_msg_trap + 10
1 libSystem.B.dylib 0x98ce6047 mach_msg + 68
2 com.apple.CoreFoundation 0x990f277f __CFRunLoopRun + 2079
3 com.apple.CoreFoundation 0x990f1864 CFRunLoopRunSpecific + 452
4 com.apple.CoreFoundation 0x990f1691 CFRunLoopRunInMode + 97
5 com.apple.Foundation 0x91b24430 +[NSURLConnection(NSURLConnectionReallyInternal) _resourceLoadLoop:] + 329
6 com.apple.Foundation 0x91aeb8d8 -[NSThread main] + 45
7 com.apple.Foundation 0x91aeb888 __NSThread__main__ + 1499
8 libSystem.B.dylib 0x98d12fbd _pthread_start + 345
9 libSystem.B.dylib 0x98d12e42 thread_start + 34

Thread 3:
0 libSystem.B.dylib 0x98d04856 select$DARWIN_EXTSN + 10
1 com.apple.CoreFoundation 0x99131ddd __CFSocketManager + 1085
2 libSystem.B.dylib 0x98d12fbd _pthread_start + 345
3 libSystem.B.dylib 0x98d12e42 thread_start + 34

Thread 4:
0 libSystem.B.dylib 0x98d0b182 __workq_kernreturn + 10
1 libSystem.B.dylib 0x98d0b718 _pthread_wqthread + 941
2 libSystem.B.dylib 0x98d0b336 start_wqthread + 30

Thread 0 crashed with X86 Thread State (32-bit):
eax: 0x00515db0 ebx: 0x990c7711 ecx: 0x00516460 edx: 0xbfffcabc
edi: 0x00001041 esi: 0x00504270 ebp: 0xbfffca38 esp: 0xbfffc99c
ss: 0x0000001f efl: 0x00010246 eip: 0x99119480 cs: 0x00000017
ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037
cr2: 0x97a20000

Wednesday, February 17, 2010

Command of the Day

ls | wc

and for when ls * fails

find . -name "*" | xargs -I {} mv {} away/

Tuesday, February 16, 2010

DS9 gaussian fitting

One thing DS9 desperately needs is an interactive gaussian fitter. I have NOT implemented one yet, but it is high on my to do list. Has anyone else (googlers?) tried or succeeded in implementing such a thing? Ideally, something with NO dependencies: if I write one, it will require python, numpy, and probably pyds9 - ick. Straight-up TCL would be very much preferable.

Another attempt to get 64 bit python on Snow Leopard

Sam Skillman posted his attempt to get 64 bit python on snow leopard. It worked, but you need to install python with --enable-framework and NOT --prefix and NOT --enable-universalSDK. It is 64-bit only, not universal: universal causes trouble.

Monday, February 15, 2010

Python 64-bit on Mac OS X 10.6 Snow Leopard

After yesterday's disastrous attempt to install various python packages, I started from scratch. First, I got rid of all of my python frameworks (backed up but removed from the path). Then, I compiled python 2.7 from scratch:

I got some help from http://blog.mahmoudimus.com/2009/12/python-2-6-4-and-twisted-9-on-os-x-10-6-snow-leopard/

./configure --enable-framework --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk MACOSX_DEPLOYMENT_TARGET=10.6 --with-universal-archs=intel -with-readline-dir=/usr/local
make -j 17
make -j 17 test


make results:
Python build finished, but the necessary bits to build these modules were not found:
_bsddb dl gdbm
imageop linuxaudiodev ossaudiodev
spwd sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


I'm not concerned about these - I don't use any of them and I assume I need to install some other packages to get them to work.

During make test, I had two failures that resulted in "python crash" pop-up boxes:
test_subprocess
.
this bit of output is from a test of stdout in a different process ...
.
this bit of output is from a test of stdout in a different process ...
test_sunaudiodev


Then, I got some malloc errors:
test_io

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.
python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
test_ioctl

Sunday, February 14, 2010

Installing Snow Leopard

I'm going to attempt to install snow leopard today. This post will serve as a record of the difficulties I run into.

Things to install (and ensure they are 64-bit):

  • ipython
  • numpy
  • scipy
  • matplotlib
  • stsci-python
  • starlink
  • gfortran
  • latex
  • idl (check)


Things that have happened:

  1. Had to restart again (twice) to install additional updates
  2. My bash command line looked funny - something about bash changed, but I don't know what. The fix was easy: commented out some code from http://pseudogreen.org/blog/set_tab_names_in_leopard_terminal.html that I had been using to set the tab title
  3. My locate db broke. Needed repair: sudo /usr/libexec/locate.updatedb
  4. numpy svn failed to build:
    python setup.py build
    Running from numpy source directory.non-existing path in 'numpy/distutils': 'site.cfg'
    F2PY Version 2_8111
    numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 4, with checksum 59750b518272c8987f02d66445afd3f1, but recorded checksum for C API version 4 in codegen_dir/cversions.txt is 3d8940bf7b0d2a4e25be4338c14c3c85. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.pyc.
    MismatchCAPIWarning)
    blas_opt_info:
    FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers']

    lapack_opt_info:
    FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-faltivec']

    running build
    running config_cc
    unifing config_cc, config, build_clib, build_ext, build commands --compiler options
    running config_fc
    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
    running build_src
    build_src
    building py_modules sources
    building library "npymath" sources
    customize NAGFCompiler
    Found executable /usr/local/bin/f95
    customize AbsoftFCompiler
    Could not locate executable f90
    Found executable /usr/bin/f77
    absoft: no Fortran 90 compiler found
    absoft: no Fortran 90 compiler found
    customize IBMFCompiler
    Could not locate executable xlf90
    Could not locate executable xlf
    customize IntelFCompiler
    Could not locate executable ifort
    Could not locate executable ifc
    customize GnuFCompiler
    Found executable /usr/local/bin/g77
    gnu: no Fortran 90 compiler found
    gnu: no Fortran 90 compiler found
    customize Gnu95FCompiler
    Found executable /usr/local/bin/gfortran
    customize Gnu95FCompiler
    customize Gnu95FCompiler using config
    C compiler: gcc -arch i386 -arch ppc -arch ppc64 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

    compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c'
    gcc: _configtest.c
    gcc _configtest.o -o _configtest
    ld: library not found for -lcrt1.10.5.o
    collect2: ld returned 1 exit status
    ld: library not found for -lcrt1.10.5.o
    collect2: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    Traceback (most recent call last):
    File "setup.py", line 210, in
    setup_package()
    File "setup.py", line 203, in setup_package
    configuration=configuration )
    File "/Users/adam/repos/numpy-svn/numpy/distutils/core.py", line 186, in setup
    return old_setup(**new_attr)
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
    File "/Users/adam/repos/numpy-svn/numpy/distutils/command/build.py", line 37, in run
    old_build.run(self)
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/command/build.py", line 134, in run
    self.run_command(cmd_name)
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/cmd.py", line 333, in run_command
    self.distribution.run_command(command)
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
    File "/Users/adam/repos/numpy-svn/numpy/distutils/command/build_src.py", line 152, in run
    self.build_sources()
    File "/Users/adam/repos/numpy-svn/numpy/distutils/command/build_src.py", line 163, in build_sources
    self.build_library_sources(*libname_info)
    File "/Users/adam/repos/numpy-svn/numpy/distutils/command/build_src.py", line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
    File "/Users/adam/repos/numpy-svn/numpy/distutils/command/build_src.py", line 385, in generate_sources
    source = func(extension, build_dir)
    File "numpy/core/setup.py", line 670, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
    RuntimeError: Broken toolchain: cannot link a simple C program


    SOLUTION: Use the Mac OS X 10.6 python (/usr/bin/python). I will do this until I run into another problem. Numpy build successfully
  5. Build/install matplotlib - failed! Completely!
  6. Acquired gcc/gfortran from hpc
  7. Followed instructions from hyperjeff on fortran install...
  8. Get rid of numpy 1.2.1: mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/.not_numpy.bak
  9. Try to install scipy. Fail on missing umfpack, follow hyperjeff's instructions (but make sure to edit the site.cfg in scipy, not just the one in numpy)

    Had to add the following code:

    sudo cp AMD/Lib/libamd.a /System/Library/Frameworks/Python.framework/Versions/2.6/lib
    sudo cp UMFPACK/Lib/libumfpack.a /System/Library/Frameworks/Python.framework/Versions/2.6/lib
    sudo cp AMD/Include/amd.h /System/Library/Frameworks/Python.framework/Versions/2.6/include
    sudo cp UFconfig/UFconfig.h /System/Library/Frameworks/Python.framework/Versions/2.6/include
    sudo cp UMFPACK/Include/*.h /System/Library/Frameworks/Python.framework/Versions/2.6/include

  10. Installed fftw from fftw.org with simple ./configure, make, sudo make install - no compiler opts as they killed the install
  11. Get SoundSource from rogueamoeba
  12. Updated istatmenus

Friday, February 12, 2010

PSDs, cross-correlation...

scipy is capable of doing fft-base cross-correlation, convolution, etc., but it requires the stsci package, which is not generally easy to install. For that matter, scipy can be a pain some of the time. So agpy now includes a 2D cross-correlation code and a power spectrum / power spectral density code. These are pure-numpy codes that should be easy to use without any other bothersome dependencies.

EDIT: I have them check for scipy (which can cause crashes if you have a bad scipy install, e.g. 32 bit executables on a 64 bit system) because scipy uses FFTW and numpy appears not to. Also, this code & related stuff has been discussed on astrobetyter

agpy
correlate2d
psds

Thursday, February 4, 2010

Logarithmic Colormap / Other Colormap in Matplotlib

This is kind of a pain to find out:


from matplotlib.colors import LogNorm

im = imshow(.... cmap=... , norm=LogNorm(vmin=clevs[0], vmax=clevs[-1]))


It also works for contours, and can be particularly useful if you only want to display contours at a few levels, but you want the colormap to start at a different point. e.g.:

contour(xx,levels=[2,3,4,5,6,7,8,9,10],norm=matplotlib.colors.Normalize(vmin=0,vmax=10))

will start at light blue instead of dark blue in the default colormap

Wednesday, January 27, 2010

wrapping text around a figure in latex

An example from Devin:

%\begin{wrapfigure}{l}{0.5\textwidth}
% \vspace{-27pt}
% \begin{center}
% \includegraphics[width=0.48\textwidth]{nsf_fig3.ps}
% \end{center}
% \vspace{-27pt}
% \caption{\it{}}
% \vspace{-12pt}
%\end{wrapfigure}
1:16
\usepackage{wrapfig}

Sunday, January 3, 2010

my month of travel

I figure most people spending a month away from home would bring more than a small duffel full of clothes. I was quite satisfied with what I had. The things I missed (as a reminder for next time):
My desktop, eta
My music collection
Powerbars

...nothin else, really