PRECIS support forum

It is currently Thu Mar 28, 2024 8:28 am

All times are UTC




Post new topic Reply to topic  [ 6 posts ] 
Author Message
PostPosted: Tue Jul 28, 2015 3:16 pm 
Offline

Joined: Tue Jul 28, 2015 12:39 pm
Posts: 19
Hi,
I am running an experiment, using ERA-Interim as boundary conditions, and everything was going perfectly , but then (10 years later) it stopped and I cannot restart the simulation (rerun to complete time). I am guessing is something with the LBC but I cannot find the error.

cheers,

Ruth

Here is part of the leave file ( i cannot attached it so i copied it here)
*****************************************************************
Job started at : lun jul 27 11:49:06 CDT 2015
Run started from UMUI
Running from control files in /home/rcerezo/umui_runs/aabab-208093502
*****************************************************************

Version 4.5 template, Unified Model , Non-Operational
Created by UMUI version 4.5.1
*****************************************************************
jobrunning file: /home/rcerezo/.jobrunning.aabab.9055
Contents:
# PIDs for PRECIS experiment aabab
9055 /home/rcerezo/umui_runs/aabab-208093502/SCRIPT

semarnat1
This job is running on machine UAS1S0P000A160U,
using UM directory /home/rcerezo/precis/um,
and test directory .
--------------------------------------------------------------------------------
PRECIS version 1.9.2

RUNNING SCRIPT WITH ARGUMENTS:

/home/rcerezo/precis/um/vn4.5/utils/restart aabab

lun jul 27 09:35:02 CDT 2015
--------------------------------------------------------------------------------
PRECIS version: 1.9.2
Max number of concurrent PRECIS jobs: 2
TYPE is CRUN in /home/rcerezo/umui_jobs/aabab/SUBMIT
FROM_PRECIS=true
Starting with command:
/home/rcerezo/precis/um/vn4.5/utils/umsubmit aabab >>/home/rcerezo/precis/tmp/aabab.restart 2>&1
Your output will be sent to file:
/home/rcerezo/precis/precis_out/aabab000.aabab.d15208.t093502.leave

warning: commands will be executed using /bin/sh
job 154 at Mon Jul 27 09:35:00 2015
--------------------------------------------------------------------------------
Contents of the SAVE_FILE:

# These are the local variables which define the experiment
# Please note that the region directory must exist
job_name: semarnat1
job_desc: test 1 para semarnat
runid: aabab
region: CORDEX-CA_EXT
rcm: HadRM3P
source: erain
idump: eraina.dai8c10
eyear: 20
emonth: 0
eday: 0
sulphur: Y
d360: N
umui_base: xacga
climate_mean: 30,3,4
cyear: 1960
cmonth: 12
cday: 1
diag_time: b
diag_lev: 850,700,500,250,50
out_format: PP
name: CORDEX-CA_EXT
dx: .44
lonlc: 303.6
latlc: 35.65
ew: 226
ns: 155
plon: 113.98
plat: 75.74
nmpp: 16
moses2: Y
--------------------------------------------------------------------------------

Restarted from rerun. Details:

--------------------------------------------------------------------------------
PRECIS version 1.9.2

RUNNING SCRIPT WITH ARGUMENTS:

/home/rcerezo/precis/um/vn4.5/utils/rerun aabab k0510 18,0,0,0,0,0 CRUN

lun jul 27 09:35:02 CDT 2015
--------------------------------------------------------------------------------
Permanent history file : /home/rcerezo/precis/precis_expt/aabab/aabab.phist

Possible restart times TYPE Restart dumps
---------------------------- ---- --------------------------------------------
k0610 = 2000/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0610
k05t0 = 2000/ 5/29 0:00:00z CRUN
k0550 = 2000/ 5/ 5 0:00:00z NRUN /home/rcerezo/precis/precis_expt/aabab/aababa.dak0550
k0510 = 2000/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0510
k0410 = 2000/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0410
k0310 = 2000/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0310
k0210 = 2000/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0210
k0110 = 2000/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0110
j9c10 = 1999/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9c10
j9b10 = 1999/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9b10
j9a10 = 1999/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9a10
j9910 = 1999/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9910
j9810 = 1999/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9810
j9710 = 1999/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9710
j9610 = 1999/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9610
j9510 = 1999/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9510
j9410 = 1999/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9410
j9310 = 1999/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9310
j9210 = 1999/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9210
j9110 = 1999/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj9110
j8c10 = 1998/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8c10
j8b10 = 1998/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8b10
j8a10 = 1998/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8a10
j8910 = 1998/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8910
j8810 = 1998/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8810
j8710 = 1998/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8710
j8610 = 1998/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8610
j8510 = 1998/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8510
j8410 = 1998/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8410
j8310 = 1998/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8310
j8210 = 1998/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8210
j8110 = 1998/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj8110
j7c10 = 1997/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7c10
j7b10 = 1997/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7b10
j7a10 = 1997/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7a10
j7910 = 1997/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7910
j7810 = 1997/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7810
j7710 = 1997/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7710
j7610 = 1997/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7610
j7510 = 1997/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7510
j7410 = 1997/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7410
j7310 = 1997/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7310
j7210 = 1997/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7210
j7110 = 1997/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj7110
j6c10 = 1996/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6c10
j6b10 = 1996/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6b10
j6a10 = 1996/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6a10
j6910 = 1996/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6910
j6810 = 1996/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6810
j6710 = 1996/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6710
j6610 = 1996/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6610
j6510 = 1996/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6510
j6410 = 1996/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6410
j6310 = 1996/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6310
j6210 = 1996/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6210
j6110 = 1996/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj6110
j5c10 = 1995/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5c10
j5b10 = 1995/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5b10
j5a10 = 1995/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5a10
j5910 = 1995/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5910
j5810 = 1995/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5810
j5710 = 1995/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5710
j5610 = 1995/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5610
j5510 = 1995/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5510
j5410 = 1995/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5410
j5310 = 1995/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5310
j5210 = 1995/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5210
j5110 = 1995/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj5110
j4c10 = 1994/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4c10
j4b10 = 1994/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4b10
j4a10 = 1994/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4a10
j4910 = 1994/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4910
j4810 = 1994/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4810
j4710 = 1994/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4710
j4610 = 1994/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4610
j4510 = 1994/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4510
j4410 = 1994/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4410
j4310 = 1994/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4310
j4210 = 1994/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4210
j4110 = 1994/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj4110
j3c10 = 1993/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3c10
j3b10 = 1993/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3b10
j3a10 = 1993/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3a10
j3910 = 1993/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3910
j3810 = 1993/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3810
j3710 = 1993/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3710
j3610 = 1993/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3610
j3510 = 1993/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3510
j3410 = 1993/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3410
j3310 = 1993/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3310
j3210 = 1993/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3210
j3110 = 1993/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj3110
j2c10 = 1992/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2c10
j2b10 = 1992/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2b10
j2a10 = 1992/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2a10
j2910 = 1992/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2910
j2810 = 1992/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2810
j2710 = 1992/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2710
j2610 = 1992/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2610
j2510 = 1992/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2510
j2410 = 1992/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2410
j2310 = 1992/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2310
j2210 = 1992/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2210
j2110 = 1992/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj2110
j1c10 = 1991/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1c10
j1b10 = 1991/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1b10
j1a10 = 1991/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1a10
j1910 = 1991/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1910
j1810 = 1991/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1810
j1710 = 1991/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1710
j1610 = 1991/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1610
j1510 = 1991/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1510
j1410 = 1991/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1410
j1310 = 1991/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1310
j1210 = 1991/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1210
j1110 = 1991/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj1110
j0c10 = 1990/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0c10
j0b10 = 1990/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0b10
j0a10 = 1990/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0a10
j0910 = 1990/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0910
j0810 = 1990/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0810
j0710 = 1990/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0710
j0610 = 1990/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0610
j0510 = 1990/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0510
j0410 = 1990/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0410
j0310 = 1990/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0310
j0210 = 1990/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0210
j0110 = 1990/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.daj0110
i9c10 = 1989/12/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9c10
i9b10 = 1989/11/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9b10
i9a10 = 1989/10/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9a10
i9910 = 1989/ 9/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9910
i9810 = 1989/ 8/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9810
i9710 = 1989/ 7/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9710
i9610 = 1989/ 6/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9610
i9510 = 1989/ 5/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9510
i9410 = 1989/ 4/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9410
i9310 = 1989/ 3/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9310
i9210 = 1989/ 2/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9210
i9110 = 1989/ 1/ 1 0:00:00z CRUN /home/rcerezo/precis/archive/aabab/dumps/aababa.dai9110
---------------------------- ---- --------------------------------------------
NRUN = Model may be restarted from this time only as an NRUN
CRUN = Model may be restarted from this time as an NRUN or a CRUN
Directories searched for restart dumps:
/home/rcerezo/precis/precis_expt/aabab
/home/rcerezo/precis/archive/aabab/dumps
/home/rcerezo/precis/archive/aabab

Copied /home/rcerezo/precis/archive/aabab/dumps/aababa.dak0510 to /home/rcerezo/precis/precis_expt/aabab
Restart dump exists : /home/rcerezo/precis/precis_expt/aabab/aababa.dak0510
Requested restart time : k0510 = 2000/ 5/ 1 0:00:00z
Length of rerun : 18,0,0,0,0,0 (Y,M,D,h,m,s)
End time : l8510 = 2018/ 5/ 1 0:00:00z
New run_target_end : 29,5,0,0,0,0 (Y,M,D,h,m,s)
New contents of /home/rcerezo/precis/precis_expt/aabab/lbc_names:
/home/rcerezo/precis/lbc/aabab/eraina.baxxxxx
$LBC_DIR/eraina.bak0510
Copy of original phist file : /home/rcerezo/precis/precis_expt/aabab/aabab.phist.k05t0.45796
Copy of original phist file : /home/rcerezo/precis/tmp/aabab.phist.k05t0.45796
New phist file for time k0510: /home/rcerezo/precis/precis_expt/aabab/aabab.phist
Copy of new phist file : /home/rcerezo/precis/precis_expt/aabab/aabab.phist.k0510
Copy of new phist file : /home/rcerezo/precis/tmp/aabab.phist.k0510
Restarting with command : /home/rcerezo/precis/um/vn4.5/utils/restart aabab
Copy of log file : /home/rcerezo/precis/precis_expt/aabab/aabab.rerun.k0510.45796
--------------------------------------------------------------------------------
TYPE=CRUN
MODEL=la365-226-155-0.44-0.44-35.65-303.6-75.74-113.98-4-19-s
TYPE=CRUN: Appending to file /home/rcerezo/precis/precis_expt/aabab/aabab.time
/bin/ksh: /home/rcerezo/umui_runs/aabab-208093502/SCRIPT[302]: cat: /home/rcerezo/precis/tmp/aabab.time: No such file or directory
rm: cannot remove ‘/home/rcerezo/precis/tmp/aabab.time’: No such file or directory
A runtime process is already running: Not running runtime
Background runtime process has PID=9087

Argument "" isn't numeric in numeric ge (>=) at /home/rcerezo/precis/um/vn4.5/scripts/ancil_update line 342.
Argument "" isn't numeric in numeric ge (>=) at /home/rcerezo/precis/um/vn4.5/scripts/ancil_update line 342.
Argument "" isn't numeric in numeric ge (>=) at /home/rcerezo/precis/um/vn4.5/scripts/ancil_update line 342.
Argument "" isn't numeric in numeric ge (>=) at /home/rcerezo/precis/um/vn4.5/scripts/ancil_update line 342.
--------------------------------------------------------------------------------
ANCILLARY FILE UPDATING

RUNNING SCRIPT WITH ARGUMENTS:
/home/rcerezo/precis/um/vn4.5/scripts/ancil_update

lun jul 27 11:49:06 CDT 2015
--------------------------------------------------------------------------------
Current model time = 2000/ 6/ 1 0:00:00z
Created soft links in /home/rcerezo/precis/precis_expt/aabab/aabab_ancil_master
to files in /home/rcerezo/precis/ancil/master
Ancillary configuration file: /home/rcerezo/precis/um/vn4.5/ctldata/ancil_config/erain
Contents:
source_ozone=ozone.echvc
source_vegsoil=qrparm.whs
source_igbp=qrdata.igbp
source_orog=qrorog.navy10.y84n
source_oxi=oxi.erain
source_sst=sst.erain
update_types=sst,sice,ozone,so2dms
source_sice=sice.erain
source_whs=qrparm.whs
source_soil=qrdata.soil
source_volc=volc.erain
source_soilcarbon=qrparm.soilcarbon
source_mask=qrfrac.navy10_BAS
source_so2dms=so2dms.erain
source_rough=qrparm.ashhr

The following ancillaries are to be updated: sst, sice, ozone, so2dms

sst : Last update time =
Current time = 2000/ 6/ 1 0:00:00z
Update time =
=> sst ancillary update required
Moved file /home/rcerezo/precis/ancil/aabab/qrclim.sst.erain to
/home/rcerezo/precis/archive/aabab/ancil/qrclim.sst.erain.k0110
New master file /home/rcerezo/precis/ancil/master/sst.erain.k0110
spans 2000/ 1/ 1 0:00:00z to
2000/12/31 18:00:00z
Soft link /home/rcerezo/precis/precis_expt/aabab/aabab_ancil_master/sst.erain ->
/home/rcerezo/precis/ancil/master/sst.erain.k0110
Next update time = 2000/ 1/ 1 0:00:00z

sice : Last update time =
Current time = 2000/ 6/ 1 0:00:00z
Update time =
=> sice ancillary update required
Moved file /home/rcerezo/precis/ancil/aabab/qrclim.sice.erain to
/home/rcerezo/precis/archive/aabab/ancil/qrclim.sice.erain.k0110
New master file /home/rcerezo/precis/ancil/master/sice.erain.k0110
spans 2000/ 1/ 1 0:00:00z to
2000/12/31 18:00:00z
Soft link /home/rcerezo/precis/precis_expt/aabab/aabab_ancil_master/sice.erain ->
/home/rcerezo/precis/ancil/master/sice.erain.k0110
Next update time = 2000/ 1/ 1 0:00:00z

ozone : Last update time =
Current time = 2000/ 6/ 1 0:00:00z
Update time =
=> ozone ancillary update required
Moved file /home/rcerezo/precis/ancil/aabab/qrclim.ozone.erain to
/home/rcerezo/precis/archive/aabab/ancil/qrclim.ozone.erain.h9bg0
New master file /home/rcerezo/precis/ancil/master/ozone.echvc.h9c10
spans 1979/11/16 0:00:00z to
2009/12/16 0:00:00z
Soft link /home/rcerezo/precis/precis_expt/aabab/aabab_ancil_master/ozone.echvc ->
/home/rcerezo/precis/ancil/master/ozone.echvc.h9c10
Next update time = 1979/11/16 0:00:00z

so2dms : Last update time =
Current time = 2000/ 6/ 1 0:00:00z
Update time =
=> so2dms ancillary update required
Moved file /home/rcerezo/precis/ancil/aabab/qrclim.so2dms.erain to
/home/rcerezo/precis/archive/aabab/ancil/qrclim.so2dms.erain.e9910
New master file /home/rcerezo/precis/ancil/master/so2dms.erain.e9c10
spans 1949/ 9/ 1 0:00:00z to
2000/ 6/ 1 0:00:00z
Soft link /home/rcerezo/precis/precis_expt/aabab/aabab_ancil_master/so2dms.erain ->
/home/rcerezo/precis/ancil/master/so2dms.erain.e9c10
Next update time = 1949/ 9/ 1 0:00:00z

Updated ancillaries will be created from master files.
Create new ancillary files:
--------------------------------------------------------------------------------
ANCILLARY FILE CREATION

RUNNING SCRIPT WITH ARGUMENTS:
/home/rcerezo/precis/um/vn4.5/utils/ancil_create -a /home/rcerezo/precis/ancil/aabab -b /home/rcerezo/precis/precis_expt/aabab/aabab_ancil_master -c /home/rcerezo/precis/um/vn4.5/ctldata/ancil_config/erain -d /home/rcerezo/umui_runs/aabab-208093502 -o /home/rcerezo/umui_runs/aabab-208093502/override_orogmask -r /home/rcerezo/umui_runs/aabab-208093502/override -t /home/rcerezo/umui_runs/aabab-208093502/override_mean_orog -u /home/rcerezo/umui_runs/aabab-208093502/override_vegsoil erain

lun jul 27 11:49:07 CDT 2015
--------------------------------------------------------------------------------
Grid namelist vn4.4 /home/rcerezo/precis/ancil/aabab/gridnamelist_vn4.4
has been created from files in /home/rcerezo/umui_runs/aabab-208093502
Grid namelist vn5.0 /home/rcerezo/precis/ancil/aabab/gridnamelist_vn5.0
has been created from /home/rcerezo/precis/ancil/aabab/gridnamelist_vn4.4
Grid namelist vn7.4 /home/rcerezo/precis/ancil/aabab/gridnamelist_vn7.4
has been created from files in /home/rcerezo/umui_runs/aabab-208093502
Ancillary configuration file: /home/rcerezo/precis/um/vn4.5/ctldata/ancil_config/erain
Contents:
source_orog=qrorog.navy10.y84n
source_sst=sst.erain
source_soil=qrdata.soil
source_so2dms=so2dms.erain
source_whs=qrparm.whs
source_ozone=ozone.echvc
source_igbp=qrdata.igbp
source_sice=sice.erain
source_volc=volc.erain
source_soilcarbon=qrparm.soilcarbon
update_types=sst,sice,ozone,so2dms
source_vegsoil=qrparm.whs
source_mask=qrfrac.navy10_BAS
source_rough=qrparm.ashhr
source_oxi=oxi.erain

mask : /home/rcerezo/precis/ancil/aabab/qrparm.mask exists. Not recreated.
/home/rcerezo/precis/ancil/aabab/qrparm.mask spans
0/ 0/ 0 0:00:00z to
0/ 0/ 0 0:00:00z
orog : /home/rcerezo/precis/ancil/aabab/qrparm.orog exists. Not recreated.
/home/rcerezo/precis/ancil/aabab/qrparm.orog spans
0/ 0/ 0 0:00:00z to
0/ 0/ 0 0:00:00z
vegsoil : Not required. Not created.
MOSES2 : /home/rcerezo/precis/ancil/aabab/qrparm.veg.frac
and/or
/home/rcerezo/precis/ancil/aabab/qrparm.veg.func
and/or
/home/rcerezo/precis/ancil/aabab/qrparm.soil.MOSES_IGBP
and/or
/home/rcerezo/precis/ancil/aabab/qrparm.veg.dist
exist. Not recreated.
/home/rcerezo/precis/ancil/aabab/qrparm.veg.frac spans
0/ 0/ 0 0:00:00z to
0/ 0/ 0 0:00:00z
/home/rcerezo/precis/ancil/aabab/qrparm.veg.func spans
0/ 0/ 0 0:00:00z to
0/ 0/ 0 0:00:00z
/home/rcerezo/precis/ancil/aabab/qrparm.soil.MOSES_IGBP spans
0/ 0/ 0 0:00:00z to
0/ 0/ 0 0:00:00z
/home/rcerezo/precis/ancil/aabab/qrparm.veg.dist spans
0/ 0/ 0 0:00:00z to
0/ 0/ 0 0:00:00z
sstice : /home/rcerezo/precis/ancil/aabab/qrclim.sice.erain
/home/rcerezo/precis/ancil/aabab/qrclim.sst.erain
created successfully.
See /home/rcerezo/precis/ancil/aabab/.aabab.sstice.out for details.
/home/rcerezo/precis/ancil/aabab/qrclim.sice.erain spans
2000/ 1/ 1 0:00:00z to
2000/12/31 18:00:00z
/home/rcerezo/precis/ancil/aabab/qrclim.sst.erain spans
2000/ 1/ 1 0:00:00z to
2000/12/31 18:00:00z
ozone : /home/rcerezo/precis/ancil/aabab/qrclim.ozone.erain
created successfully.
See /home/rcerezo/precis/ancil/aabab/.aabab.ozone.out for details.
/home/rcerezo/precis/ancil/aabab/qrclim.ozone.erain spans
1979/11/16 0:00:00z to
2009/12/16 0:00:00z
so2dms : /home/rcerezo/precis/ancil/aabab/qrclim.so2dms.erain
created successfully.
See /home/rcerezo/precis/ancil/aabab/.aabab.so2dms.out for details.
/home/rcerezo/precis/ancil/aabab/qrclim.so2dms.erain spans
1949/ 9/ 1 0:00:00z to
2000/ 6/ 1 0:00:00z
oxi : /home/rcerezo/precis/ancil/aabab/qrclim.oxi.erain exists. Not recreated.
/home/rcerezo/precis/ancil/aabab/qrclim.oxi.erain spans
1990/ 1/16 0:00:00z to
1990/12/16 0:00:00z
volc : /home/rcerezo/precis/ancil/aabab/qrparm.volc.erain exists. Not recreated.
/home/rcerezo/precis/ancil/aabab/qrparm.volc.erain spans
1991/ 1/ 1 0:00:00z to
1991/ 1/ 1 0:00:00z

Final contents of ANCIL_DIR= /home/rcerezo/precis/ancil/aabab
total 1664044
drwxrwxr-x 2 rcerezo 4096 jul 27 11:49 .
drwxrwxr-x 5 rcerezo 4096 jul 9 15:49 ..
-rw-rw-r-- 1 rcerezo 62058 jul 9 15:49 .aabab.mask.out
-rw-rw-r-- 1 rcerezo 913340 jul 9 15:51 .aabab.MOSES2.out
-rw-rw-r-- 1 rcerezo 334776 jul 9 15:49 .aabab.orog.out
-rw-rw-r-- 1 rcerezo 198445 jul 9 15:52 .aabab.oxi.out
-rw-rw-r-- 1 rcerezo 762843 jul 27 11:49 .aabab.ozone.out
-rw-rw-r-- 1 rcerezo 197073 jul 27 11:49 .aabab.so2dms.out
-rw-rw-r-- 1 rcerezo 845446 jul 27 11:49 .aabab.sstice.out
-rw-rw-r-- 1 rcerezo 114564 jul 9 15:52 .aabab.volc.out
-rw-rw-r-- 1 rcerezo 357 jul 27 11:49 gridnamelist_vn4.4
-rw-rw-r-- 1 rcerezo 324 jul 27 11:49 gridnamelist_vn5.0
-rw-rw-r-- 1 rcerezo 255 jul 27 11:49 gridnamelist_vn7.4
-rw-rw-r-- 1 rcerezo 656 jul 9 15:49 .make_ancil_mask.namelist
-rwxr-xr-x 1 rcerezo 1699 jul 9 15:49 .make_ancil_mask.script
-rw-rw-r-- 1 rcerezo 2018 jul 9 15:49 .make_ancil_MOSES2.namelist
-rw-rw-r-- 1 rcerezo 5213 jul 9 15:49 .make_ancil_MOSES2.script
-rw-rw-r-- 1 rcerezo 812 jul 9 15:49 .make_ancil_orog.namelist
-rwxr-xr-x 1 rcerezo 1880 jul 9 15:49 .make_ancil_orog.script
-rw-rw-r-- 1 rcerezo 504 jul 9 15:52 .make_ancil_oxi.namelist
-rw-rw-r-- 1 rcerezo 1650 jul 9 15:52 .make_ancil_oxi.script
-rw-rw-r-- 1 rcerezo 504 jul 27 11:49 .make_ancil_ozone.namelist
-rw-rw-r-- 1 rcerezo 1656 jul 27 11:49 .make_ancil_ozone.script
-rw-rw-r-- 1 rcerezo 504 jul 27 11:49 .make_ancil_so2dms.namelist
-rw-rw-r-- 1 rcerezo 1659 jul 27 11:49 .make_ancil_so2dms.script
-rw-rw-r-- 1 rcerezo 657 jul 27 11:49 .make_ancil_sstice.namelist
-rwxr-xr-x 1 rcerezo 1828 jul 27 11:49 .make_ancil_sstice.script
-rw-rw-r-- 1 rcerezo 504 jul 9 15:52 .make_ancil_volc.namelist
-rw-rw-r-- 1 rcerezo 1653 jul 9 15:52 .make_ancil_volc.script
-rw-rw-r-- 1 rcerezo 459 jul 9 15:49 .overridefile_mask
-rw-rw-r-- 1 rcerezo 134717440 jul 9 15:52 qrclim.oxi.erain
-rw-rw-r-- 1 rcerezo 1015971840 jul 27 11:49 qrclim.ozone.erain
-rw-rw-r-- 1 rcerezo 216245080 jul 27 11:49 qrclim.sice.erain
-rw-rw-r-- 1 rcerezo 120537088 jul 27 11:49 qrclim.so2dms.erain
-rw-rw-r-- 1 rcerezo 216245080 jul 27 11:49 qrclim.sst.erain
-rw-rw-r-- 1 rcerezo 148312 jul 9 15:49 qrparm.mask
-rw-rw-r-- 1 rcerezo 1033048 jul 9 15:49 qrparm.orog
-rw-rw-r-- 1 rcerezo 148312 jul 9 15:49 qrparm.orog.mask
-rw-rw-r-- 1 rcerezo 1630208 jul 9 15:51 qrparm.soil.MOSES_IGBP
-rw-rw-r-- 1 rcerezo 155648 jul 9 15:51 qrparm.veg.dist
-rw-rw-r-- 1 rcerezo 1335296 jul 9 15:51 qrparm.veg.frac
-rw-rw-r-- 1 rcerezo 1482752 jul 9 15:51 qrparm.veg.func
-rw-rw-r-- 1 rcerezo 2809856 jul 9 15:52 qrparm.volc.erain
--------------------------------------------------------------------------------
Ancillary creation completed at lun jul 27 11:49:39 CDT 2015
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Ancillary updating completed at lun jul 27 11:49:39 CDT 2015
--------------------------------------------------------------------------------
Time taken by /home/rcerezo/precis/um/vn4.5/scripts/ancil_update:
**ANCIL_TIME: 0m33.14s real 0m27.67s user 0m5.44s system


--------------------------------------------------------------------------------
LATERAL BOUNDARY CONDITION UPDATING

RUNNING SCRIPT WITH ARGUMENTS:
/home/rcerezo/precis/um/vn4.5/scripts/lbc_update

lun jul 27 11:49:39 CDT 2015
--------------------------------------------------------------------------------
TYPE = CRUN
Restart time = 2000/ 6/ 1 0:00:00z
Next update time = 2000/ 6/ 1 0:00:00z (k0610)
LBC file /home/rcerezo/precis/lbc/aabab/eraina.bak0610 exists and has non-zero size.
Deleted LBC file /home/rcerezo/precis/lbc/aabab/eraina.bak0610
Creating LBC file /home/rcerezo/precis/lbc/aabab/eraina.bak0610. Details:
Using standard prognostics PP4LBC file /home/rcerezo/precis/pp4lbc/erain/eraina.CORDEX_CentralAmerica.pik0610
See log file /home/rcerezo/precis/precis_expt/aabab/lbc/eraina.bak0610.out
LBC file /home/rcerezo/precis/lbc/aabab/eraina.bak0610 exists and has non-zero size.
Created soft link:
/home/rcerezo/precis/precis_expt/aabab/alabcin_aabab -> /home/rcerezo/precis/lbc/aabab/eraina.bak0610
Deleted LBC file: /home/rcerezo/precis/lbc/aabab/eraina.bak0510
Current and future LBC files:
/home/rcerezo/precis/lbc/aabab/eraina.bak0610
$LBC_DIR/eraina.bak0710
--------------------------------------------------------------------------------
Lateral boundary condition updating completed at lun jul 27 11:49:41 CDT 2015
--------------------------------------------------------------------------------
Time taken by /home/rcerezo/precis/um/vn4.5/scripts/lbc_update:
**LBC_TIME: 0m1.69s real 0m0.78s user 0m0.70s system

PATH USED = /home/rcerezo/precis//cdat-3.3/bin:/home/rcerezo/precis/um/ui:/home/rcerezo/precis/um/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rcerezo/precis/um/vn4.5/utils:/home/rcerezo/precis/um/bin/mpich2/bin:/home/rcerezo/precis/um/rove:/home/rcerezo/precis/tmp/modscr_aabab:/vn4.5/exec::/u/m20/home/mec/t20sm/rove:/home/rcerezo/precis/um/vn4.5/scripts:/home/rcerezo/precis/um/vn4.5/exec
*****************************************************************
Starting script : qsprelim
Starting time : lun jul 27 11:49:41 CDT 2015
*****************************************************************


/home/rcerezo/precis/um/vn4.5/scripts/qsprelim: Executing combine

/home/rcerezo/precis/um/vn4.5/scripts/qscombine: Job terminated normally
qsprelim: /home/rcerezo/precis/precis_expt/aabab/aabab.thist_keep moved to /home/rcerezo/precis/precis_expt/aabab/aabab.thist
*****************************************************************
Ending script : qsprelim
Completion code : 0
Completion time : lun jul 27 11:49:41 CDT 2015
*****************************************************************



*****************************************************************
Starting script : qsmain
Starting time : lun jul 27 11:49:41 CDT 2015
*****************************************************************

aabab: Starting run
hello -machinefile /home/rcerezo/precis/tmp/machines.208093502 -np 16 /home/rcerezo/precis/precis_expt/aabab/aabab.parexe

=====================================================
GCOM Version 3.0
MPI
Using precision : 32bit INTEGERs and 32bit REALs
Built at Wed Apr 30 10:34:12 BST 2008
=====================================================

OPEN: File /home/rcerezo/precis/precis_expt/aabab/aababa.dak0610 to be Opened on Unit 21 Exists
OPEN: File /home/rcerezo/precis/precis_expt/aabab/aababa.dak0610 to be Opened on Unit 21 Exists
OPEN: File /home/rcerezo/precis/ancil/aabab/qrclim.ozone.erain to be Opened on Unit 30 Exists
OPEN: File /home/rcerezo/precis/ancil/aabab/qrclim.sst.erain to be Opened on Unit 35 Exists
OPEN: File /home/rcerezo/precis/ancil/aabab/qrclim.sice.erain to be Opened on Unit 36 Exists
OPEN: File /home/rcerezo/precis/ancil/aabab/qrclim.so2dms.erain to be Opened on Unit 110 Exists
OPEN: File /home/rcerezo/precis/ancil/aabab/qrclim.oxi.erain to be Opened on Unit 116 Exists
OPEN: File /home/rcerezo/precis/precis_expt/aabab/alabcin_aabab to be Opened on Unit 95 Exists
OPEN: File /home/rcerezo/precis/precis_expt/aabab/aababa.pak0may to be Opened on Unit 60 Exists
OPEN: File /home/rcerezo/precis/precis_expt/aabab/aababa.pjk0610 to be Opened on Unit 69 Exists
*****************************************
*****************************************
Model completed with the following :
Error Code : 539
Message : REPLANCA: TIME INTERPOLATION ERROR
*****************************************
*****************************************
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
forrtl: error (76): IOT trap signal
Aborted (core dumped)
rank 5 in job 143 UAS1S0P000A160U_47705 caused collective abort of all ranks
exit status of rank 5: killed by signal 9
0m1.24s real 0m0.07s user 0m0.01s system
qsmain: Time taken by model executable /home/rcerezo/precis/um/vn4.5/exec/hadrm3p.MOSESII.exec.mpp:
**MODEL_TIME:
qsmain: LOADMODULE CC=137

diff: /home/rcerezo/precis/tmp/aabab.xhist: No such file or directory
qsmain: Copying /home/rcerezo/precis/precis_expt/aabab/aabab.thist to backup thist file /home/rcerezo/precis/precis_expt/aabab/aabab.thist_keep
aabab: Run failed
aabab: Information on which CPU was in use follows:-

aabab: debug information (if activated) follows :-

/home/rcerezo/precis/um/vn4.5/scripts/qsmain[1010]: debugview: not found
core file copied to /home/rcerezo/precis/tmp/aabab.core
*****************************************************************
Ending script : qsmain
Completion code : 137
Completion time : lun jul 27 11:49:44 CDT 2015
*****************************************************************





/home/rcerezo/precis/um/vn4.5/scripts/qsmaster: failed in main in model aabab
*****************************************************************
Starting script : qsfinal
Starting time : lun jul 27 11:49:44 CDT 2015
*****************************************************************

qsfinal: thist file copied to /home/rcerezo/precis/precis_expt/aabab/aabab.thist.9427
/home/rcerezo/precis/um/vn4.5/scripts/qsfinal: Error in exit processing after model run
Failed in model executable

/home/rcerezo/precis/um/vn4.5/scripts/qspickup: Normal completion
/home/rcerezo/precis/um/vn4.5/scripts/qshistprint: Job terminated normally
/home/rcerezo/precis/um/vn4.5/scripts/qsresubmit: Error job not resubmitted because of error in qsmain
*****************************************************************
Ending script : qsfinal
Completion code : 135
Completion time : lun jul 27 11:49:45 CDT 2015
*****************************************************************





/home/rcerezo/precis/um/vn4.5/scripts/qsmaster: failed in final in model aabab
<<<< Information about How Many Lines of Output follow >>>>
7997 lines in main OUTPUT file.
22 lines in SERVER file.
<<<< Lines of Output Information ends >>>>
OUTPUT
5394 lines of output from pe0.
PE0 OUTPUT
%PE0 OUTPUT%
16 Processors initialised.
I am PE 0
Start of UM Job : 11:49:42 on 27/07/2015
Setting UM_NAM_MAX_SECONDS to 300
UMSETUP; CRUN, read CONTCNTL from unit 9
Number of levels for convective clouds is 19
global_land_field set to 10321
PE 0 : LAND_FIELD is being reset from 10321 to 1505
local_land_field set to 1505
n_intf_a 0
n_intf_a 1
tot_len_intfa_p 1
tot_len_intfa_u 1
max_intf_p_levels 1
u_field_intfa 1
!!!! STASH_MSTR
/home/rcerezo/precis/um/vn4.5/ctldata/STASHmaster/STASHmaster_A

MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 229
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 230
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 231
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 3 Item 314
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 3 Item 321
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 3 Item 322
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 3 Item 287
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 3 Item 288
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 101
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 102
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 103
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 104
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 105
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 106
MESSAGE FROM ROUTINE GETPPX:
The following PPXREF record has been overwritten by
a record read from a user-STASH master file:
Internal Model 1 Section 0 Item 107
PRELIM: START TIME BEFORE PERIOD, SETTING TO 1
MODEL,SECTION,ITEM 1 1 201
PRELIM: START TIME BEFORE PERIOD, SETTING TO 1
MODEL,SECTION,ITEM 1 1 235
PRELIM: START TIME BEFORE PERIOD, SETTING TO 1
MODEL,SECTION,ITEM 1 2 201
PRELIM: START TIME BEFORE PERIOD, SETTING TO 1
MODEL,SECTION,ITEM 1 2 204
MESSAGE FROM ROUITNE PRELIM: DIAGNOSTIC REQUEST
HAS LEVEL RANGE OUT OF BOUNDS; HAS BEEN CORRECTED
MODEL,SECTION,ITEM 1 3 219
MESSAGE FROM ROUTINE PRELIM: DIAGNOSTIC REQUEST
HAS TOP/BOTTOM LEVELS INCONSISTENT; REQUEST IGNORED
MODEL,SECTION,ITEM 1 3 219
MESSAGE FROM ROUITNE PRELIM: DIAGNOSTIC REQUEST
HAS LEVEL RANGE OUT OF BOUNDS; HAS BEEN CORRECTED
MODEL,SECTION,ITEM 1 3 220
MESSAGE FROM ROUTINE PRELIM: DIAGNOSTIC REQUEST
HAS TOP/BOTTOM LEVELS INCONSISTENT; REQUEST IGNORED
MODEL,SECTION,ITEM 1 3 220
MESSAGE FROM ROUTINE PRELIM: DIAGNOSTIC REQUEST HAS
INVALID PSEUDO LEVEL TYPE; REQUEST IGNORED
MODEL,SECTION,ITEM 1 3 293
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01001218
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003274
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003275
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003276
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003277
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003278
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003279
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003280
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003281
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01003286
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01005207
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01005208
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01005222
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01005223
: FILE EXISTS
OPEN: 3:
/home/rcerezo/precis/um/vn4.5/ctldata/stasets/X01010206
: FILE EXISTS
ADDRES : A_PROG_LOOKUP = 488
ADDRES : A_PROG_LEN = 1179656
ADDRES : S_PROG_LOOKUP = 0
ADDRES : S_PROG_LEN = 0
ADDRES : O_PROG_LOOKUP = 0
ADDRES : O_PROG_LEN = 0
ADDRES : W_PROG_LOOKUP = 0
ADDRES : W_PROG_LEN = 0
285 D1 items in submodel 1
323 STASH LIST RECORDS



Free disk space:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda4 4054032664 776648104 3071428516 21% /home

****************************************************************
Job ended at : lun jul 27 11:49:45 CDT 2015
****************************************************************
Killed the background runtime process with PID=9087

runtime output at end of run:
------------------------------------------------------+-------------------------
REAL TIME : 11:49:45 CDT lunes 27/jul/2015 | RATE : 1467588.25 months/day
MODELS : HadRM3P (MOSES2.2), ERA-Interim (erain) | TIME : 13 seconds (end)
RUNID : aabab | DATE : 11:49 27/Jul/2015
STATUS : CRUN finished at 11:49 27/Jul/2015 | MWLE : 0%
------------------------------+-----------------------+-------------------------
start : 1988/12/ 1 0:00z | COMPLETED : 11 Years 5 Months 28 Days
CURRENT : 2000/ 5/29 0:00z | % : 39.067% of 29 Years 5 Months
end : 2018/ 5/ 1 0:00z | TIMESTEPS : 2417472/6187968 (150 secs/TS)
------------------------------+-------------------------------------------------

MWLE, timestep and time information from /home/rcerezo/precis/precis_expt/aabab/aabab.time:
# Col 1: Output file
# Col 2: Model time at start of file
# Col 3: Model time at end of file
# Col 4: No. of MWLE timesteps in output file
# Col 5: No. of MWLE timesteps in whole run
# Col 6: No. of timesteps in output file
# Col 7: First timestep in output file
# Col 8: Last timestep in output file
# Col 9: Number of real seconds for CRUN
# Col 10: Number of real seconds for run so far
/home/rcerezo/precis/precis_out/aabab000.aabab.d15190.t154917.leave , 1988/12/ 1 , 1989/ 1/ 1 , 0 , 0 , 8928 , 1 , 8928 , 3 , 3
/home/rcerezo/precis/precis_out/aabab000.aabab.d15191.t103545.leave , 1989/ 1/ 1 , 1989/ 2/ 1 , 0 , 0 , 8928 , 8929 , 17856 , 0 , 3
/home/rcerezo/precis/precis_out/aabab000.aabab.d15195.t152248.leave , 1993/ 2/ 1 , 1993/ 3/ 1 , 0 , 0 , 8064 , 438625 , 446688 , 0 , 3
/home/rcerezo/precis/precis_out/aabab000.aabab.d15208.t093250.leave , 1988/12/ 0 , 1988/12/ 1 , 0 , 0 , 0 , 0 , 0 , 0 , 3
/home/rcerezo/precis/precis_out/aabab000.aabab.d15208.t093502.leave

Executing exit commands:
kill -9 9087 ; rm -f /home/rcerezo/.jobrunning.aabab.9055 /home/rcerezo/precis/tmp/aabab.runtime_info /home/rcerezo/precis/tmp/aabab.runtime_rate
/bin/ksh: kill: 9087: No such process


Top
 Profile  
 
PostPosted: Mon Aug 03, 2015 12:10 pm 
Offline

Joined: Thu Mar 27, 2014 5:03 pm
Posts: 73
Hi Ruth,

Did you try using the rerun command after the model had already crashed?

I don't know if this is relevant, but in more recent PRECIS versions we have disabled the CRUN option from the rerun command because it was error prone. Are you able to try submitting as an NRUN? i.e. home/rcerezo/precis/um/vn4.5/utils/rerun aabab k0510 18,0,0,0,0,0 NRUN , and if the model successfully runs for a month restart aabab?

All the best,
Simon

_________________
Simon Tucker
Met Office Hadley Centre


Top
 Profile  
 
PostPosted: Thu Aug 27, 2015 5:28 pm 
Offline

Joined: Tue Jul 28, 2015 12:39 pm
Posts: 19
Hi!

I restarted the full simulation, since I got a new ERA-Interim dataset (starting in 1979), and everything run smoothly until the simulation reached May 2000.
It is sending the same error message: REPLANCA: TIME INTERPOLATION ERROR

I have tried to restart the model from 1st May 2000 (with rerun k0510...NRUN)
and it did run for the full month then stop (as normal) if I type: restart idrun, the model tried to restart but then crashed with the same error

then I tried to restart from 1st June 2000(since the model have that start dump), again using rerun k0610 ...NRUN
and it crashed
and with CRUN the same...
I attached here the last leave file...help!


Attachments:
aabad000.aabad.d15239.t122028.leave.gz [71.72 KiB]
Downloaded 335 times
Top
 Profile  
 
PostPosted: Wed Sep 02, 2015 2:17 pm 
Offline

Joined: Thu Mar 27, 2014 5:03 pm
Posts: 73
Dear Ruth,

To restart your simulation again, you simply need to unzip the file $ANCIL_MASTER/so2dms.erain.k0310.gz

I am afraid however that an error has been found in the SO2DMS ancillaries that you are using. Specifically the DMS fields contain no emissions over the oceans. It is unlikely that the error will have a large affect on your simulation, however please contact us should you wish to rerun with a corrected ancillary.

Regards,
Simon

_________________
Simon Tucker
Met Office Hadley Centre


Top
 Profile  
 
PostPosted: Thu Sep 03, 2015 2:57 pm 
Offline

Joined: Tue Jul 28, 2015 12:39 pm
Posts: 19
Hi Simon,

Thanks for the reply, but I am not so sure that I understood.
Do you mean that all the DMS files that have been used on this simulation have an error ? including this one that I have just unzip?

I am no so sure I will re-do the simulation now, but it will be good to have the right files (just in case) and probably now that I have to restart the simulation to use the right one?
How can I get the correct one?

cheers,
Ruth


Top
 Profile  
 
PostPosted: Wed Sep 09, 2015 10:20 am 
Offline

Joined: Thu Mar 27, 2014 5:03 pm
Posts: 73
Hi Ruth,


Sorry, I thought that I replied to your message on Monday.

Replacement ancillaries can be downloaded from http://www.precisrcm.com/Ruth/. To use these you will need to edit the file $UMDIR/vn4.5/ctldata/ancil_config/erain by replacing the line
Quote:
source_so2dms = so2dms.erain
with
Quote:
source_so2dms = so2dms.historical.greg

_________________
Simon Tucker
Met Office Hadley Centre


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC


Who is online

Users browsing this forum: Google [Bot] and 51 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group