fr.ifremer.isisfish.simulator.launcher

Class SSHSimulatorLauncher

java.lang.Object

fr.ifremer.isisfish.simulator.launcher.SSHSimulatorLauncher

All Implemented Interfaces:

SimulatorLauncher

public class SSHSimulatorLauncher
extends Object
implements SimulatorLauncher

Use a remote simulation server. Upload zip simulation file on server and launch simulation on that file. Isis-Fish must be installed on remote server. Caparmor file layout ($i = plan/as increment) :

• $ISIS-TMP/simulation-$id-preparation.zip
• $ISIS-TMP/simulation-$id-prescript.bsh
• $ISIS-TMP/simulation-$id-script.seq
• $ISIS-TMP/simulation-$id-result.zip
• $ISIS-TMP/simulation-$id-result.zip.md5
• $ISIS-TMP/simulation-$id-output.txt
• $ISIS-TMP/simulation-$id-pbs.id

All $ISIS-TMP/simulation-$id-* files are deteled after result download. Special case :

• standalone zip : $ISIS-TMP/simulation-$shortid-result.zip (uploaded at first simulation)
• standalone simulations : $ISIS-TMP/simulation-$shortid-script.seq (uploaded at last simulation)

where $shortid is id of parent job (without increment), they are not deleted after result download.

Version:

$Revision: 4017 $ Last update : $Date: 2014-06-16 18:04:46 +0200 (Mon, 16 Jun 2014) $ By : $Author: echatellier $

Author:

chatellier

See Also:

JSch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SSHSimulatorLauncher.ControlProgressMonitor Redefine a custom progress monitor that update control.

Field Summary

Fields

Modifier and Type	Field and Description
protected freemarker.template.Configuration	freemarkerConfiguration Freemarker configuration
protected static org.apache.commons.logging.Log	log Class logger
protected static String	QSUB_SCRIPT_TEMPLATE Freemarker qsub template.
protected static com.jcraft.jsch.Session	sshSession Opened session to ssh service.
protected static com.jcraft.jsch.Session	sshSftpSession Opened session to sftp service.

Constructor Summary

Constructors

Constructor and Description
SSHSimulatorLauncher() Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	clearSimulationFiles(com.jcraft.jsch.Session session, SimulationControl control) Remove all $ISIS-TMP/simulation-$id-* files on caparmor.
protected void	closeSSHSession(com.jcraft.jsch.Session session) Close ssh session.
protected File	downloadResultsArchive(com.jcraft.jsch.Session session, SimulationControl simulationControl, String md5sum) Download simulation zip results.
protected File	downloadResultsMD5File(com.jcraft.jsch.Session sshSession, String simulationId) Download remote simulation md5 control file and store its content into temp file.
protected File	downloadSimulationFile(com.jcraft.jsch.Session sshSession, String simulationId, String fileName) Download remote simulation control file and store its content into temp file.
int	getCheckProgressionInterval() Retourne l'intervalle de temps a utiliser entre deux verification de progresssion.
protected File	getLaunchSimulationScriptFile(String simuationId, String simulationZip, boolean standaloneZip, String simulationResultZip, String preScriptPath, boolean multipleSimulationScript) Retourne un fichier temporaire contenant le script de lancement de simulation.
protected String	getRemoteResultArchivePath(String simulationId) Get remote simulation zip path.
protected String	getRemoteTempDirectory() Get remote directory absolute path.
protected String	getSimulationScriptLaunchContent(String templateName, String simuationId, String simulationZip, boolean standaloneZip, String simulationZipResult, String preScriptPath, boolean multipleSimulationScript) Utilise freemarker pour recuperer le contenu du script.
SimulationStorage	getSimulationStorage(SimulationService simulationService, SimulationControl control) Retourne le storage où est stockée la simulation.
protected com.jcraft.jsch.Session	getSSHSession() Get opened ssh session or try to open a new one.
protected void	initFreemarker() Init freemarker configuration.
int	maxSimulationThread() Retourne le nombre maximal de thread de simulations simultanées supporté.
protected void	message(SimulationControl control, String message) Display message both in commons-logging and control text progress.
protected com.jcraft.jsch.Session	openSSHSession() Connect to remote server throw SSH, and return session.
protected void	putSshKeyOnRemoteServer(com.jcraft.jsch.Session session, File sshKey) Add ssh key into $HOME/.ssh/authorized_keys file.
protected void	sendStartSimulationRequest(com.jcraft.jsch.Session session, String simulationId, String scriptRemotePath, int lastSimulationNumber) Add script in remote qsub queue.
protected void	sendStopSimulationRequest(com.jcraft.jsch.Session session, String simulationId) Send qdel request on job.
void	simulate(SimulationService simulationService, SimulationItem simulationItem) Execute une simulation.
void	simulationStopRequest(SimulationJob job) Called if a started simulation has been ask to stop.
protected void	startSimulation(com.jcraft.jsch.Session session, SimulationItem simulationItem, String simulationid, String simulationRemoteZipPath, String remoteResultZip, String simulationPreScriptPath) Start simulation if necessary.
String	toString()
void	updateControl(SimulationService simulationService, SimulationControl control) Met à jour la progression.
protected String	uploadPreScriptIfNecessary(com.jcraft.jsch.Session session, String simulationId, String simulationPreScript) Upload pre script on remote server.
protected void	uploadSimulation(com.jcraft.jsch.Session session, String remoteDirectory, String remoteSimulationZipPath, File simulationFile) Perform simulation upload.
protected String	uploadSimulationIfNecessary(com.jcraft.jsch.Session session, SimulationItem simulationItem, String simulationid, File simulationFile) Upload simulation if necessary and always return the remote simulation zip path to use.
protected String	uploadSimulationScript(com.jcraft.jsch.Session session, String simulationid, File simulationScript) Upload script on remote server.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

log

protected static org.apache.commons.logging.Log log

Class logger

freemarkerConfiguration

protected freemarker.template.Configuration freemarkerConfiguration

Freemarker configuration

QSUB_SCRIPT_TEMPLATE

protected static final String QSUB_SCRIPT_TEMPLATE

Freemarker qsub template.

See Also:

Constant Field Values

sshSession

protected static com.jcraft.jsch.Session sshSession

Opened session to ssh service. Stored in static context to not reask passphrase at each connection.

sshSftpSession

protected static com.jcraft.jsch.Session sshSftpSession

Opened session to sftp service. Stored in static context to not reask passphrase at each connection.

Constructor Detail

SSHSimulatorLauncher

public SSHSimulatorLauncher()

Constructor. Init freemarker.

Method Detail

initFreemarker

protected void initFreemarker()

Init freemarker configuration.

message

protected void message(SimulationControl control,
           String message)

Display message both in commons-logging and control text progress.

Parameters:

control - control

message - message to display

maxSimulationThread

public int maxSimulationThread()

Description copied from interface: SimulatorLauncher

Retourne le nombre maximal de thread de simulations simultanées supporté.

Specified by:

maxSimulationThread in interface SimulatorLauncher

Returns:

un nombre superieur a 0

getCheckProgressionInterval

public int getCheckProgressionInterval()

Description copied from interface: SimulatorLauncher

Retourne l'intervalle de temps a utiliser entre deux verification de progresssion.

Specified by:

getCheckProgressionInterval in interface SimulatorLauncher

Returns:

intervalle (en secondes);

toString

public String toString()

Overrides:

toString in class Object

simulationStopRequest

public void simulationStopRequest(SimulationJob job)
                           throws RemoteException

Called if a started simulation has been ask to stop. Tell launcher to stop simulation if possible. Try to send a qdel command.

Specified by:

simulationStopRequest in interface SimulatorLauncher

Parameters:

job - job to stop

Throws:

RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte.

simulate

public void simulate(SimulationService simulationService,
            SimulationItem simulationItem)
              throws RemoteException

Execute une simulation. Dans le cas de ssh:

• upload la simulation
• construit le script pour qsub
• upload le script qsub
• ajoute le script a qsub

Et :

• lance le thread de control de la simulation

Specified by:

simulate in interface SimulatorLauncher

Parameters:

simulationService - le SimulationService qui a initie la simulation

simulationItem - l'item contenant toutes les infos nécessaire au lancement de la simulation IMPORTANT Quoi qu'il arrive

Throws:

RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte. Dans ce cas la simulation n'a pas ete faite et cette exception est levee.

getSimulationStorage

public SimulationStorage getSimulationStorage(SimulationService simulationService,
                                     SimulationControl control)
                                       throws RemoteException

Retourne le storage où est stockée la simulation. Se connecte au serveur distant et télécharge les résultats de la simulation. Simulation must have been downloaded with updateControl(SimulationService, SimulationControl) before calling this method.

Specified by:

getSimulationStorage in interface SimulatorLauncher

Parameters:

simulationService - le SimulationService qui a initie la simulation

control - le control de la simulation

Returns:

le storage contenant la simulation qui vient d'etre faite ou null en fait au lieu de faire une seul simulation, plusieurs ou ete faite par exemple tout un plan de simulation Cette classe est tout de meme responsable du stockage de chaque simulation dans des storages.

Throws:

RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte.

updateControl

public void updateControl(SimulationService simulationService,
                 SimulationControl control)
                   throws RemoteException

Met à jour la progression. Se connecte au serveur distant et télécharge le fichier de control. Injecte ensuite ce fichier dans le SimulationControl. Essaye aussi de telecharger le fichier md5 de la simulation, et, s'il est present, l'archive de résultat. Supprime tous les fichiers de la simulations apres avoir télécharger les résultats.

Specified by:

updateControl in interface SimulatorLauncher

Parameters:

simulationService - le SimulationService qui a initie la simulation

control - le control de la simulation

Throws:

RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte.

getSSHSession

protected com.jcraft.jsch.Session getSSHSession()
                                         throws com.jcraft.jsch.JSchException

Get opened ssh session or try to open a new one. This method must synchronized.

Returns:

opened ssh session.

Throws:

com.jcraft.jsch.JSchException

openSSHSession

protected com.jcraft.jsch.Session openSSHSession()
                                          throws com.jcraft.jsch.JSchException

Connect to remote server throw SSH, and return session.

Returns:

valid opened session

Throws:

com.jcraft.jsch.JSchException

closeSSHSession

protected void closeSSHSession(com.jcraft.jsch.Session session)

Close ssh session.

Parameters:

session - session to close

putSshKeyOnRemoteServer

protected void putSshKeyOnRemoteServer(com.jcraft.jsch.Session session,
                           File sshKey)
                                throws com.jcraft.jsch.JSchException

Add ssh key into $HOME/.ssh/authorized_keys file. Just connect and do an "echo xx >> .ssh/authorized_keys"

Parameters:

session - opened session

sshKey -

Throws:

com.jcraft.jsch.JSchException

uploadSimulationIfNecessary

protected String uploadSimulationIfNecessary(com.jcraft.jsch.Session session,
                                 SimulationItem simulationItem,
                                 String simulationid,
                                 File simulationFile)
                                      throws SSHException

Upload simulation if necessary and always return the remote simulation zip path to use.

Parameters:

session - already open valid ssh session

simulationItem - simulation item

simulationid - simulation id

simulationFile - simulation file to upload

Returns:

remote file path or null if errors

Throws:

SSHException - if upload fail

uploadSimulation

protected void uploadSimulation(com.jcraft.jsch.Session session,
                    String remoteDirectory,
                    String remoteSimulationZipPath,
                    File simulationFile)
                         throws SSHException

Perform simulation upload. Create remote temp directory if not exists.

Parameters:

session - already open valid ssh session

remoteDirectory -

remoteSimulationZipPath -

Throws:

SSHException

downloadResultsArchive

protected File downloadResultsArchive(com.jcraft.jsch.Session session,
                          SimulationControl simulationControl,
                          String md5sum)
                               throws SSHException,
                                      IOException

Download simulation zip results. MD5 control check sum if done, return null, if checkSum fail. File if configured to auto delete at JVM shutdown.

Throws:

SSHException - if download fail (can happen if remote file doesn't exist

IOException - if download fail (can happen if remote file doesn't exist

downloadSimulationFile

protected File downloadSimulationFile(com.jcraft.jsch.Session sshSession,
                          String simulationId,
                          String fileName)
                               throws IOException,
                                      SSHException

Download remote simulation control file and store its content into temp file.

Parameters:

sshSession - valid opened ssh session

simulationId - id de la simulation

fileName - nom du fichier a telecharger

Returns:

downloaded temp file (file have to be manually deleted)

Throws:

IOException

SSHException - if remote file doesn't exists

downloadResultsMD5File

protected File downloadResultsMD5File(com.jcraft.jsch.Session sshSession,
                          String simulationId)
                               throws IOException,
                                      SSHException

Download remote simulation md5 control file and store its content into temp file.

Parameters:

sshSession - valid opened ssh session

simulationId - id de la simulation

Returns:

downloaded temp file (file have to be manually deleted)

Throws:

IOException

SSHException - if remote file doesn't exists

clearSimulationFiles

protected void clearSimulationFiles(com.jcraft.jsch.Session session,
                        SimulationControl control)
                             throws IOException,
                                    SSHException

Remove all $ISIS-TMP/simulation-$id-* files on caparmor.

Parameters:

session - valid opened ssh session

control - simulation control

Throws:

IOException

SSHException

uploadSimulationScript

protected String uploadSimulationScript(com.jcraft.jsch.Session session,
                            String simulationid,
                            File simulationScript)
                                 throws SSHException

Upload script on remote server.

Parameters:

session - valid opened ssh session

simulationScript - file to upload

Throws:

SSHException - if upload fail

getRemoteResultArchivePath

protected String getRemoteResultArchivePath(String simulationId)

Get remote simulation zip path.

Parameters:

simulationId - simulation id

uploadPreScriptIfNecessary

protected String uploadPreScriptIfNecessary(com.jcraft.jsch.Session session,
                                String simulationId,
                                String simulationPreScript)
                                     throws SSHException,
                                            IOException

Upload pre script on remote server. Return path if uploaded or null if no upload needed.

Parameters:

session - valid opened ssh session

simulationId - simulation id

simulationPreScript - script content

Throws:

SSHException - if upload fail

IOException - if upload fail

startSimulation

protected void startSimulation(com.jcraft.jsch.Session session,
                   SimulationItem simulationItem,
                   String simulationid,
                   String simulationRemoteZipPath,
                   String remoteResultZip,
                   String simulationPreScriptPath)
                        throws Exception

Start simulation if necessary. Current simulation can be started later with a PBS multi job.

Parameters:

session - ssh session

simulationItem - simulation item (needed for additionnal info, simulation, number, indenpendant, etc...)

simulationid - simulation id

simulationRemoteZipPath - simulation preparation (input) zip path

remoteResultZip - simulation result (output) zip path

simulationPreScriptPath - simulation prescript

Throws:

Exception

getLaunchSimulationScriptFile

protected File getLaunchSimulationScriptFile(String simuationId,
                                 String simulationZip,
                                 boolean standaloneZip,
                                 String simulationResultZip,
                                 String preScriptPath,
                                 boolean multipleSimulationScript)
                                      throws IOException

Retourne un fichier temporaire contenant le script de lancement de simulation. Le fichier temporaire est configuré pour se supprimer tout seul.

Parameters:

simuationId - id de la simulation

simulationZip - zip de la simulation

standaloneZip - standalone simulation zip

preScriptPath - simulation pre script path (can be null)

multipleSimulationScript - if true build a multijob simulation script

Returns:

un Fichier temporaire ou null en cas d'exception

Throws:

IOException - if can't build script

getSimulationScriptLaunchContent

protected String getSimulationScriptLaunchContent(String templateName,
                                      String simuationId,
                                      String simulationZip,
                                      boolean standaloneZip,
                                      String simulationZipResult,
                                      String preScriptPath,
                                      boolean multipleSimulationScript)
                                           throws IOException

Utilise freemarker pour recuperer le contenu du script. Remplace aussi la variable $simulation du template.

Parameters:

templateName - url du template

simuationId - id de la simulation

simulationZip - zip de la simulation

standaloneZip - standalone simulation zip

simulationZipResult - zip resultat de la simulation

preScriptPath - simulation pre script path (can be null)

multipleSimulationScript - if true build a multijob simulation script

Throws:

IOException - if can't get script content

sendStartSimulationRequest

protected void sendStartSimulationRequest(com.jcraft.jsch.Session session,
                              String simulationId,
                              String scriptRemotePath,
                              int lastSimulationNumber)
                                   throws SSHException

Add script in remote qsub queue.

Parameters:

session - valid opened session

simulationId - simulation id (short version for a multiple job)

scriptRemotePath - remote script path

lastSimulationNumber - if >=0 start a multiple pbs job form 0 to lastSimulationNumber

Throws:

SSHException - if call fail

sendStopSimulationRequest

protected void sendStopSimulationRequest(com.jcraft.jsch.Session session,
                             String simulationId)
                                  throws SSHException

Send qdel request on job.

Parameters:

session - valid opened session

simulationId - simulation id

Throws:

SSHException - if call fail

getRemoteTempDirectory

protected String getRemoteTempDirectory()

Get remote directory absolute path. Don't use File.separator here, caparmor is always unix.

Returns:

remote temp directory path