Howtodoit.txt

*) PREREQUISITES:

python2.6
python modules:
	lxml
	mx.DateTime or egenix_mx_base
	rdflib 3
	pysolr
	simplejson

(Use pip or easy_install)

*) BASIC INSTALL:

mkdir ads-stuff
git clone git@github.com:rahuldave/semflow.git
rsync -avz ~/Dropbox/AstroExplorer .

If Installing AppSem:
Install node.js 0.4.8
install modules using npm (which therefore also needs to be installed):
	connect connect-redis hiredis mime mustache redis qs
	
git clone git@github.com:rahuldave/appsem.git
cd appsem
git submodule init
git submodule update

In appsem run node server.js and point at 
    http://localhost:3000/semantic2/alpha/explorer/publications/

or open
   appsem/static/ajax-solr/publications.html
in your browser (this will only work if you have the webapp up and
running on port 8983, as shown below).

*) VENDOR INSTALLS:

mkdir ads-stuff/vendor
cd ads-stuff/vendor

(a) Install Solr
tar zxvf ../AstroExplorer/vendor/apache-solr-3.2.0.tgz
cp -a apache-solr-3.2.0/example container

(b) Install Sesame
tar zxvf ../AstroExplorer/vendor/openrdf-sesame-2.4.0-sdk.tar.gz
cp openrdf-sesame-2.4.0/war/openrdf-* container/webapps/

(c) Install Redis
tar zxvf ../AstroExplorer/vendor/redis-2.2.10.tar.gz
cd redis-2.2.10
make; sudo make install
/usr/local/bin/redis-server (if installed)
redis-cli can be used to check it

(d) Start Sesame and Solr on port 8983:
pushd container/solr/conf
mv solrconfig.xml solrconfig.xml.bak
mv schema.xml schema.xml.bak
# Link to configuration files in semflow repository
ln -s ../../../../semflow/solr/schema.xml .
ln -s ../../../../semflow/solr/solrconfig.xml .
popd
cd container
#Now back in container directory
java -jar start.jar
#With more memory for heap
-Xmx4g

Solr admin is at:
http://localhost:8983/solr/admin/
Seasme Workbench is at:
http://localhost:8983/openrdf-workbench/

*) CREATION OR CLEANING OF THE DATABASES:

We have three data stores that either need creating or cleaning:

(a) on disk storage

This is in the chandra-rdf/ and mast-rdf/ directories created in the
SET UP FOR DATA INSTALL step below.

(b) Sesame database

This can be created (or deleted and then created) using either the
Sesame workbench interface at
  http://localhost:8983/openrdf-workbench/

or using the console.sh script at
  vendor/openrdf-sesame-2.4.0/bin/console.sh

The repository should have
   Type = Native Java Store
   Id     = testads8
   Title = testads8

and accept the default value for the triple indexes (which will be
"spoc,posc".

Here's a run-through using the command-line interface, which is
described at
  http://www.openrdf.org/doc/sesame2/users/ch07.html

% vendor/openrdf-sesame-2.4.0/bin/console.sh
03:11:25.843 [main] DEBUG info.aduna.platform.PlatformFactory - os.name = mac os x
03:11:25.848 [main] DEBUG info.aduna.platform.PlatformFactory - Detected Mac OS X platform
Connected to default data directory

Commands end with '.' at the end of a line
Type 'help.' for help
> connect http://localhost:8983/openrdf-sesame.
Disconnecting from default data directory
Connected to http://localhost:8983/openrdf-sesame
> show repositories.
+----------
|SYSTEM ("System configuration repository")
+----------
> create native.
Please specify values for the following variables:
Repository ID [native]: testads8
Repository title [Native store]: testads8
Triple indexes [spoc,posc]: 
Repository created
> show repositories.
+----------
|SYSTEM ("System configuration repository")
|testads8 ("testads8")
+----------
> quit.
Disconnecting from http://localhost:8983/openrdf-sesame
Bye

The database should be deleted and re-created if re-running the
scripts. This can be done from the web page or from within the console
by saying

  connect http://localhost:8983/openrdf-sesame.
  drop testads8.
  create native.
  ...

You may want to try physically deleting the files from disk after the
drop but before re-creating: on OS-X this worked for me:

rm -rf ~/Library/Application\ Support/Aduna/OpenRDF\ Sesame/repositories/testads8/

(c) Solr database

It looks like there's no need to create a database for Solr. If
re-running the scripts then the existing data needs to be removed
which is achieved with the solrclear.py script:

  cd semflow
  python solrclear.py

There is also

  python solropt.py

which optimizes the Solr database; this is mainly useful to reduce the number of open
files (so allowing more data to be added) but it may improve search times somewhat.

*) SETUP FOR DATA INSTALL:

Depending on if u r on a mac or nor , the directory will be Chandra or CHANDRA. Make sure one is
symbolically linked to the other. as below on linux

cd ads-stuff
tar zxvf AstroExplorer/Missions/Chandra/chandra/chda.tgz
pushd AstroExplorer/Missions/
ln -s Chandra CHANDRA
popd
mkdir chandra-rdf
mkdir mast-rdf

cd semflow
export PYTHONPATH=`pwd`

POSSIBLE THINGIES NEEDING DOING
(a) java heap size
(b) ulimits on ubuntu

*) CHANDRA INSTALL:

To do everything (there are also scripts for the other missions but
these are not included in this document); the scripts log information
about each stage to <mission>.log (to make it easier to see how long
stages are taking and to see if any fail).

% ./scripts/doitchandra.sh

Manual steps are:

adsrdf:
    python adsclassic2rdf.py ../chandra-rdf ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.biblist.txt
    python adsclassic2rdf.py ../chandra-rdf ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.biblist.txt
simbadrdf:
    python simbad2rdf.py ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.simbad.dict ../chandra-rdf
    python simbad2rdf.py ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.simbad.dict ../chandra-rdf
pubrdf:
    python chandra/genrdf.py pub ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.linkedpubs.txt ../chandra-rdf/
    python chandra/genrdf.py pub ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.linkedpubs.txt ../chandra-rdf/
obsvrdf:
    python chandra/genrdf.py obsv ../AstroExplorer/Missions/Chandra/chandra/global.obsids.txt ../chandra-rdf/
proprdf:
    python chandra/genrdf.py prop ../AstroExplorer/Missions/Chandra/chandra/global.proposals.txt ../chandra-rdf/
    
adsload:    
    python loadfiles.py ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.biblist.txt
    python loadfiles.py ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.biblist.txt
simbadload:
    python loadfiles-simbad.py ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.biblist.txt
    python loadfiles-simbad.py ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.biblist.txt
pubload:
    python chandra/loadfiles.py ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.linkedpubs.txt pub
    python chandra/loadfiles.py ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.linkedpubs.txt pub

obsvload:
    python chandra/loadfiles.py ../AstroExplorer/Missions/Chandra/chandra/global.obsids.txt obsv
propload:
    python chandra/loadfiles.py ../AstroExplorer/Missions/Chandra/chandra/global.proposals.txt prop
    
#We had to produce a cut file below due to some linkage probs in Chandra
pubsolr:
    #python solrclear.py #is doing this first
    python rdf2solr5.py CHANDRA chandra ../AstroExplorer/Missions/Chandra/chandra/sherry.p.a.biblist.txt.cut 
    python rdf2solr5.py CHANDRA chandra ../AstroExplorer/Missions/Chandra/chandra/hutoverlap.biblist.txt

MAST:

For each mission there needs to be a file
  mast/ingest_<mission>.py

  newmast/mast_proprdf_<mission>.py
    (only if there are proposals for the mission)

In most cases this can be simple, such as mast/ingest_wuppe.py, but it
can get complex (e.g. hut).

In order to match bibcodes, datasets and possibly proposals, you need
to understand how the obsid value in the map.<mission>.txt file
relates to the obsid values from the obscore.<mission>.psv file. In
most cases the map file just gives the prefix of the obscore value
(hopefully a unique prefix) but there are cases where more
manipulation is needed (e.g. case conversion), which should be done by
the getObsidForPubMap routine. This routine takes in the obsid value which we
use to create the URI for the object, so essentially the obscore
value, and converts it to a form that can be compared to the map
version via 
  if getObsidForPubMap(...).startswith(obsid-from-map-file):
    start processing

Note that Doug has messed around with this flow slightly and now made
the comparison case insensitive, so you should not need to do case
conversion in the ingest_<mission>.py file, but this change may be
backed out.

Ok, doing HUT
--------------



1992BAAS...24.1285L
1992AAS...18110208L
BAAS error, need to use the ADS bibcode synonym file
No output, need to fix first.
Multiple fixes were needed. We corrected at file level by new alts.py in scripts folder.
also see errors.txt in hut where we had to exscise one non-existent bibcode

adsrdf:
	python adsclassic2rdf.py ../mast-rdf ../AstroExplorer/Missions/MAST/hut/hut.biblist.txt
simbadrdf:
	python simbad2rdf.py ../AstroExplorer/Missions/MAST/hut/hut.simbad.dict ../mast-rdf
	
# Order is important here	
obsvrdf:
	python newmast/mast_obsvrdf.py hut ../AstroExplorer/Missions/MAST/hut/obscore.hut.psv 	
pubrdf:

	python newmast/mast_pubrdf.py hut ../AstroExplorer/Missions/MAST/hut/map.hut.txt
proprdf:
	echo None

adsload:
	python loadfiles.py ../AstroExplorer/Missions/MAST/hut/hut.biblist.txt default2.conf
	
simbadload:
	python loadfiles-simbad.py ../AstroExplorer/Missions/MAST/hut/hut.biblist.txt default2.conf
	
obsvload:
	python newmast/mast_obsvload.py hut
	
pubload:
	python newmast/mast_pubload.py hut

propload:
	echo None
	
pubsolr:
	python rdf2solr5.py MAST hut ../AstroExplorer/Missions/MAST/hut/hut.biblist.txt


WUPPE

adsrdf:
	python adsclassic2rdf.py ../mast-rdf ../AstroExplorer/Missions/MAST/wuppe/wuppe.biblist.txt
simbadrdf:
	python simbad2rdf.py ../AstroExplorer/Missions/MAST/wuppe/wuppe.simbad.dict ../mast-rdf
	
# Order is important here	
obsvrdf:
	python newmast/mast_obsvrdf.py wuppe ../AstroExplorer/Missions/MAST/wuppe/obscore.wuppe.psv 	
pubrdf:

	python newmast/mast_pubrdf.py wuppe ../AstroExplorer/Missions/MAST/wuppe/map.wuppe.txt
proprdf:
	echo None

adsload:
	python loadfiles.py ../AstroExplorer/Missions/MAST/wuppe/wuppe.biblist.txt default2.conf
	
simbadload:
	python loadfiles-simbad.py ../AstroExplorer/Missions/MAST/wuppe/wuppe.biblist.txt default2.conf
	
obsvload:
	python newmast/mast_obsvload.py wuppe
	
pubload:
	python newmast/mast_pubload.py wuppe

propload:
	echo None
	
pubsolr:
	python rdf2solr5.py MAST wuppe ../AstroExplorer/Missions/MAST/wuppe/wuppe.biblist.txt
	


HPOL

adsrdf:
	python adsclassic2rdf.py ../mast-rdf ../AstroExplorer/Missions/MAST/hpol/hpol.biblist.txt
	
simbadrdf:
	python simbad2rdf.py ../AstroExplorer/Missions/MAST/hpol/hpol.simbad.dict ../mast-rdf
	
# Order is important here	
obsvrdf:
	python newmast/mast_obsvrdf.py hpol ../AstroExplorer/Missions/MAST/hpol/obscore.hpol.psv 	
	
pubrdf:
	python newmast/mast_pubrdf.py hpol ../AstroExplorer/Missions/MAST/hpol/map.hpol.txt
	
proprdf:
	echo None

adsload:
	python loadfiles.py ../AstroExplorer/Missions/MAST/hpol/hpol.biblist.txt default2.conf
	
simbadload:
	python loadfiles-simbad.py ../AstroExplorer/Missions/MAST/hpol/hpol.biblist.txt default2.conf
	
obsvload:
	python newmast/mast_obsvload.py hpol
	
pubload:
	python newmast/mast_pubload.py hpol

propload:
	echo None
	
pubsolr:
	python rdf2solr5.py MAST hpol ../AstroExplorer/Missions/MAST/hpol/hpol.biblist.txt	
	
	
EUVE

adsrdf:
	python adsclassic2rdf.py ../mast-rdf ../AstroExplorer/Missions/MAST/euve/euve.biblist.txt
	
simbadrdf:
	python simbad2rdf.py ../AstroExplorer/Missions/MAST/euve/euve.simbad.dict ../mast-rdf
	
# Order is important here	
obsvrdf:
	python newmast/mast_obsvrdf.py euve ../AstroExplorer/Missions/MAST/euve/obscore.euve.psv 
		
pubrdf:
	python newmast/mast_pubrdf.py euve ../AstroExplorer/Missions/MAST/euve/map.euve.txt
	
proprdf:
	python newmast/mast_proprdf.py euve ../AstroExplorer/Missions/MAST/euve/euve_program.list

adsload:
	python loadfiles.py ../AstroExplorer/Missions/MAST/euve/euve.biblist.txt default2.conf
	
simbadload:
	python loadfiles-simbad.py ../AstroExplorer/Missions/MAST/euve/euve.biblist.txt default2.conf
	
obsvload:
	python newmast/mast_obsvload.py euve
	
pubload:
	python newmast/mast_pubload.py euve

propload:
	python newmast/mast_propload.py euve
	
pubsolr:
	python rdf2solr5.py MAST euve ../AstroExplorer/Missions/MAST/euve/euve.biblist.txt
	

FUSE


adsrdf:
	python adsclassic2rdf.py ../mast-rdf ../AstroExplorer/Missions/MAST/fuse/fuse.biblist.txt
	
simbadrdf:
	python simbad2rdf.py ../AstroExplorer/Missions/MAST/fuse/fuse.simbad.dict ../mast-rdf
	
# Order is important here	
obsvrdf:
	python newmast/mast_obsvrdf.py fuse ../AstroExplorer/Missions/MAST/fuse/obscore.fuse.psv 
		
pubrdf:
	python newmast/mast_pubrdf.py fuse ../AstroExplorer/Missions/MAST/fuse/map.fuse.txt
	
proprdf:
	python newmast/mast_proprdf.py fuse ../AstroExplorer/Missions/MAST/fuse/fuse_program.list

adsload:
	python loadfiles.py ../AstroExplorer/Missions/MAST/fuse/fuse.biblist.txt default2.conf
	
simbadload:
	python loadfiles-simbad.py ../AstroExplorer/Missions/MAST/fuse/fuse.biblist.txt default2.conf
	
obsvload:
	python newmast/mast_obsvload.py fuse
	
pubload:
	python newmast/mast_pubload.py fuse

propload:
	python newmast/mast_propload.py fuse
	
pubsolr:
	python rdf2solr5.py MAST fuse ../AstroExplorer/Missions/MAST/fuse/fuse.biblist.txt