| blob | ae0a0e008f264ac3940d3b8456f9a7e3aec9c326 |
1 This version of PsN is tested with perl-5.8.8
2 Depending on your distribution you may have to install the following
3 perl packages:
5 Math::Random
6 Storable::Store
8 All available from CPAN ( www.cpan.org )
10 Changelog
11 ----------------------------------------------
13 Changes from 2.2.4 to 2.2.5
15 Features
17 - mcs script renamed to sse
18 - sse calculates lots of nice results.
20 Bugs
22 - Multiline ETABAR and PVAL parsed correctly.
24 - Modelfit did not copy extra subroutine files in some cases.
26 - <new script> Add IGNORE=@ to estimation models unless it
27 exists. Necessary for ignoring headers in table output.
29 - Memory leak in output parser plugged. Output files from simulations
30 could use vast amounts of memory.
32 - Bootstrap options "skip_minimization_terminated" can now be disabled
33 by typing "-no-skip-minimization_terminated" (or -noskip-...)
35 - cdd xv can no be disabled with -no-xv or -noxv
37 - mc tool no uses considerably less memory. It is a lot slower
38 because it reads a lot of unecessary stuff from disk, it should be
39 reviewed.
41 - Memory leak in modelfit due to intermediate "raw_results.csv" in
42 NM_runX directory fixed by removing the feature, should be removed.
44 - A initial estimate of form: $OMEGA BLOCK SAME would not be
45 recognized by PsN model file parser. This is now handled
46 correctly. In 2.2.X it can be avoided by typing BLOCK(X) SAME,
47 where X is the size of the block.
49 Changes from 2.2.3 to 2.2.4
51 Features
53 - "--compute_cwres" option renamed to simply "--cwres"
55 - Simulated data suitable for creating mirror plots can be generated
56 by using a new options called --mirror_plots. The option takes a
57 value which defines the number of simulations that will be
58 geneated.
60 - A new option, --iofv, enables the computation of individual
61 objective function values. These values are printed to a table file
62 called iotab*, where * is the same number as is defined for any
63 sdtab or patab tables. These names correspond to the normal Xpose
64 table file format. If no sdtab or patab table is found in the
65 NONMEM control stream, the iotab file is given number 1.
67 - msfo resumes. Basic support for automatically rerunning a model
68 that has the MSFO=msffile option on $ESTIMATION. If the option is
69 set, the "msffile" exists and the option --msfo_resume is given,
70 PsN will remove initial estimates, and add the $MSFI record. This
71 is very untested and if you have OMEGA blocks, the model will run
72 but PsN will be unable to read the output file properly.
74 - "no_remote_compile" and "no_remote_execution" options has been
75 removed, as they were quite seldomly used and added a lot of
76 complexity to parallization code.
78 - Added --prepend_model_file_name option which adds the model file
79 name to output files to prevent them from being overwritten.
81 - A new option called "crash_restarts" which controls the number of
82 times PsN will restart a crashed run. (Simply put, run is
83 classified as crashed if the output file looks like it is cut off)
85 - PsN will now automatically run R-scripts generated for the
86 bootstrap, cdd and llp if R is installed and configured.
88 - When the CONT data item is used - either defined by the user or
89 automatically through the -wrap_data option - the tables get a
90 weird format where the real table content is printed on CONT=0 rows
91 and nonsense data printed on CONT=1 rows. A new option called
92 -unwrap_table_files reformats the table files and removes the CONT
93 column as well as all CONT=1 rows.
95 - update_inits script that will put final estimates from a NONMEM
96 output file and put into a given model file. This is quite
97 experimental. PsN will reformat the model file quite lot and
98 comments in the file might get lost. Your model file will be copied
99 with the addition of a ".org" extension, so you wont loose your
100 model.
102 Bug Fixes
104 - SGE queue and resource options were ignored. Fixed thanks to Jeroen
105 Elassaiss and Stefan Verhoeven.
107 - LSF Monitor fixed to work with the new serialized structure of PsN,
108 again thanks to Jeroen and Stefan.
110 - CWRES table file number now defaults to 1 if there is no patab or
111 sdtab to take the number from.
113 - Fixed run bugs under Windows 2000.
115 - Better support for G77 under windows.
117 - Parser for output files improved
119 Changes from 2.2.2 to 2.2.3
121 Features
123 - There has been a lot of requests for a raw_results file for the
124 runs in the NM_run catalogs, i.e. a summary of the runs with
125 perturbed initial estimates. This is now implemented. One
126 raw_results file is now created in each NM_run* catalog.
128 - The R-scripts for graphical visualization of bootstrap and cdds
129 have been updated. The format for the graphs is now pdf instead of
130 postscript. The bootstrap script can now be adjusted to exclude
131 bootstrap samples that contain copies of a given set of
132 individuals.
134 - When the wrap_data option is used, the secondary columns, i.e. the
135 data columns that are kept on rows with CONT=1, are now available
136 in $ERROR.
138 Bug Fixes
140 - The calculation of CWRES was previously done on the _last_ run of a
141 set of retries, not on the _best_ run. This has now been corrected.
143 - The restarting of crashed runs was broken. This is now fixed.
145 - The asynchronuous job scheduling broke the handling of seed numbers
146 for random sequences (e.g. perturbation of initial estimates). This
147 is now fixed. The present version is however not compatible with
148 older versions and old runs should therefore not be resumed using
149 this version.
151 - wrap_data should now work as it should. The functionality was
152 broken as a consequence of the move to the asynchronuous job
153 handling.
155 - Labels for omegas were not handled correctly for SAME blocks. Among
156 other things, this had the effect that the updating of initial
157 estimates from the original model to the method created models
158 (e.g. bootstrap samples) was corrupted. This is now fixed.
160 - The transformation of the estimate of the omega matrix from
161 variance and covariances to standard deviations and correlations
162 was rewritten to address problems with a combination of BLOCK(X)
163 ans SAME where X was larger than 1.
165 - The selection of a 'best' model fit after a series of perturbations
166 of the initial estimates will be changed. In 2.2.2, the lowest OFV
167 among fits of the perturbed models was collected across all models
168 (still only comparing identical models with perturbed initial
169 estimates). In 2.2.3 the lowest OFV is collected in three
170 categories of model fits: those that minimized succesfully, those
171 that did not minimize successfully but returned an OFV plus an
172 estimate of the number of significant digits and finally failed
173 runs that managed to return an OFV but not much more. This will
174 give PsN a more fine-grained control over the results. For example,
175 it would be desirable to be able to return the run with the lowest
176 OFV within the selection of runs with a successful minimization
177 even though the lowest OFV of all perturbed runs was returned from
178 a run that terminated.
180 - The sigmas were not parsed correctly when tables were printed to
181 the output file. This is not fixed.
183 - Terminated covariance steps were sometimes still being parsed (and
184 failed). Fixed.
186 - Setting the clean level to 3 did not remove the
187 NM_run-directories. This is now fixed.
189 - The parsing of options for the initial values of omegas and sigmas
190 sometimes wrongly recognized letters behind a semi-colon (i.e. a
191 comment) as keywords for a block (FIX, SAME, etc). This is now
192 corrected.
194 - If you "fix" more than one omega or sigma in a block PsN would put
195 a FIX after each value, which NONMEM doesn't allow. Now PsN will
196 only put one FIX inside a block. However, if a any part of the
197 block has been fixed, its likely so that PsN can not(!) unfix
198 it.(so, one bug fixed, another introduced)
200 - If you an initial estimate was added with "_init_attr", the option
201 "add_if_absent" was true and the last initial estimate was "SAME"
202 PsN would try to add the estimate after "SAME" which is wrong, now
203 PsN adds a new record.
205 - Fixed a parsing error of the lst file. If simulation without a
206 SIGMA was performed, the parser failed to read OMEGA initial
207 estiamtes.
209 - The internal method "indexes" would not properly handle "SAME" in
210 initial estimates. The result was that "update_inits" would skip
211 initial values, and get them out of order. Fixed.
214 Changes from 2.2.2-beta4 to 2.2.2
216 Features
218 - CWRES can now be calculated automatically if you have R and Xpose
219 installed. Remember to add a definition of IPRED in your NONMEM model.
221 - This version handles non-PsN standard (or absent) suffixes for
222 model files better than earlier versions.
224 Bug Fixes
226 - MSFO files are now copied back to the same directory of the model
227 file.
229 - MSFI files are now automatically copied to the PsN NONMEM run
230 directories.
232 - Output files from runs where the $MSFI record was used were not
233 parsed correctly. This is now fixed.
235 - Output files from runs where no $OMEGA _or_ $SIGMA were defined
236 were not parsed correctly.
238 - The header and the results from problematic runs, i.e. failed
239 covariance step, warnings etc, were not handled correctly when the
240 raw_results files were compiled. This is now fixed.
242 - In 2.2.1, we added a check for a '1' to find the end of the omega
243 and sigma estimate areas. this was a mistake since problems with
244 more than 13 omegas or sigmas get a '1'-line inserted before the
245 14:th estimate (for some reason). This is now removed.
247 Changes from 2.2.2-beta3 to 2.2.2-beta4
249 Features
251 - No new features
253 Bug Fixes
255 - Adaptive number of threads is now working in the new asynchronuous
256 job handling
258 - The CWRES computation had some fixes
260 - $INFN is now printed before $MODEL
262 - $OMEGA is now printed befor $PK
264 - PRIOR is now handled in the output module. It is still a bit
265 problematic to parse PRIOR output files correctly. Use with
266 causion.
268 - The initial estimates of large omega and sigma matrices is now
269 better handled
271 - The order of parsing events in the output module is now working
272 better.
274 - The handling of retries (or the results of retries) is now working
275 as it should.
277 Changes from 2.2.2-beta2 to 2.2.2-beta3
279 Features
281 - CWRES: Conditional weighted residuals. The necessary verbatim code
282 for the computation of CWRES are now automatically added if the
283 option -compute_cwres is used.
285 - PRIOR: The use of priors in NONMEM VI is now supported in the
286 computer intensive methods
288 - The behavior of the parallel execution has been revised, moving
289 from a synchronuous to an asynchronuous solution. Previously, the
290 memory foot-print of a parallel execution of say 100 NONMEM jobs
291 was huge (counted in several GB) and did not scale well. Now, it
292 will very seldom exceed 100 MB and more or less keep that size,
293 regardless of the number of parallel jobs.
295 Bug Fixes
297 - PsN couldn't parse IGNORE=(ITEM=VALUE) options in $DATA. This is
298 (again) fixed. It was previously corrected in the development
299 branch of PsN but never transferred to the stable branch.
301 - Fixed SAME block correlation computation.
303 Changes from 2.2.2-beta1 to 2.2.2-beta2
305 Features
307 - The parsing of the NONMEM output files has been revised and a
308 strict check is now performed after each section has been parsed. A
309 parsing error is raised if any non-compliant structures are
310 found. By doing this we get a better idea of which runs that really
311 finishes and which runs that are pre-maturely terminated
312 (e.g. through a operating system error). Remember that this is a
313 beta release and that there could still be valid text structures
314 that are not yet recognised correctly by PsN.
316 - Support for United Devices grip MP has been added and is available using
317 "-run_on_ud" command line option.
319 Bug Fixes
321 - the options directory and backwards_dir can now be specified in the
322 scm configuration file
324 - It is now possible to skip the definition of either continuous and
325 categorical covariates in the scm configuration file
327 - The exponential relation is now added correctly in the
328 scm. Previously it was incorrectly added as (for weight on volume)
329 TVV = THETA(1)*(1+EXP(THETA(2)*(WGT-median(WGT))))
330 Now it is added as
331 TVV = THETA(1)*EXP(THETA(2)*(WGT-median(WGT)))
333 - The output parse now computes correlations for omegas and sigmas correctly
334 when you have SAME blocks.
336 Changes from 2.2.1 to 2.2.2-beta1
338 Features
340 - The number of significant digits per parameter after a terminated
341 minimization are now parsed.
343 Bug Fixes
345 - The sample size option of the bootstrap did not work correctly with
346 stratified resampling. This is now fixed.
349 Changes from 2.2.0 to 2.2.1
351 Features
353 - Added support for the Intel Fortran compiler (mostly for those
354 using the new Intel Macs).
356 - We have added two new model fit quality terms: 'terminated' and
357 'really_bad' to the previous 'normal' and 'picky'.
359 picky - Minimization and covariance steps successful, no
360 parameters near boundaries and no warnings from the
361 covariance step.
363 normal - Minimization successful
365 terminated - Estimates of the objective function value and the
366 number of significant digits are available.
368 really_bad - An estimate of the OFV is available (but usually not
369 much more...).
371 Given that more than one run from a run has been performed (with
372 different initial estimates but otherwise identical) the following
373 selection procedure is done:
375 The lowest OFV of all tries is collected. Then, all runs within
376 5 units of the lowest OFV is tried according to
378 1. Of the runs passing the picky criteria, choose the run with the
379 highest number of significant digits.
381 2. If no run has passed the criteria above: of the runs passing the
382 normal criteria, choose the one with the highest number of
383 sginificant digits.
385 3. If no run has passed the criteria above: of the runs passing the
386 terminated criteria, choose the one with the highest number of
387 sginificant digits.
389 4. If still no run has passed the criteria above: of all runs that
390 at least have an estimate of the OFV, choose the run with the
391 lowest OFV.
393 5. As a last resort, choose the first run.
395 Comments: This is of course a very pragmatic approach but it is
396 needed to make the automatic procedures of the PsN tools to
397 work. We cannot force the user to manually scrutinise the results
398 of every single run of for example a bootstrap. The user has the
399 opportunity to do so if he or she wishes. If you do not feel
400 comfortable with this selection procedure or if you have comments
401 on how to improve it, please contact me (Lars Lindbom) at
402 lars.lindbom (at) farmbio.uu.se.
404 Bugfixes
406 - Data files with spaces around commas used for columns separation
407 could be problematic to parse. This is now fixed.
409 - There was a serious error in the seed handling in version
410 2.2.0. Under the following circumstances, the seed number was
411 ignored and a default seed number used: If the output file from the
412 original run was not present in the same directory as the original
413 model file when a PsN tool was started. This affected all PsN
414 tools, except the execute tool. This is now fixed.
416 - Up to 20 omegas can now be parsed in the output file. This is still
417 a limitation and it will be addressed in later releases.
419 - On fast distributed systems with a shared file system, the massive
420 parallelisation of the PsN tools could cause synchronization
421 problems (files could sometimes not be found, read or written). The
422 root of this problem is somewhat outside of the scope of PsN but we
423 have tried to find a workaround for this for the LSF grid system.
425 Changes from 2.1.10 to 2.2.0 ( released for PsN course )
427 Features
429 - PsN Output module use a proper Matrix library for handling some of
430 the matrices of the outputfile.
432 - The bootstrap can now be resumed with a lower number of
433 samples. Meaning that if you are in a hurry and want to stop the
434 bootstrap and get results after 1000 out of 2000 runs, you can. At
435 a later time you can then resume the 2000 runs.
437 - Nonmem warnings are now printed if "--verbose" is givven.
439 - Automatic incrementation of directory names are a bit smarter (no
440 more "directory modelfit_dirX is in the way" messages).
442 - Clean is much more stringent and intelligent and now manages to
443 remove entire tool directories if desired.
445 - option list printed from "-h|-?" only shows utility specific options.
447 - Default values configurable on a per utility basis.
449 - Online documentation improved
451 - Parallel run of NONMEM is not aborted as before if a outputfile is
452 not correctly parsed. It is just ignored, as if it had not
453 minimized successfully. (actually it is the output parser that
454 doesn't die, but the effect is the same)
456 - Parallel run is a bit nicer to the system when compiling many
457 NONMEMS. By adding a configurable delay between starts.
459 - Matlab and R scripts for various plots from cdd and bootstrap
460 results are distributed with PsN.
462 - Vast improvments in results presentations from all tools.
464 - Improvements in rerun algoritm with new "min_retries" option.
466 - Output from running utilties are much nicer.
468 - new "-quick_summary" prints minimization message and ofv value for
469 each finished run. As does "-verbose" option which also prints
470 "R:X" for each retrie (X is the retrie number).
472 - new "-summary" options prints diagnostics about the tool run when
473 it is finished. For "execute" the "-summary" option will have the
474 same function as the "sumo" utiltiy as well as some diagnostics.
476 - PsN now recognizes that control files may require fortran files and
477 copies them to the run directory. No need to add fotran files to
478 the --extra_files option any more.
480 - Added --extra_output which allows you to copy more than the output
481 file from the run directory.
483 Bugs
485 - PsN couldn't parse IGNORE=(ITEM=VALUE) options in $DATA. Fixed :)
487 - If star notation was used in the scm config file section [inits],
488 [upper_bounds] or [lower_bounds] there was a risk of defining
489 relations that was not specified in [test_relations]. Which in turn
490 would cause a lot of diffrent errors. Fixed :)
492 - Using low_cutoff and hi_cutoff in data::sd resulted in PsN trying to use
493 the new compact data strings as the old arrays. fixed.
495 - Removed unecessary regular expression in output file parser.
497 - Not really a bug. But i tried to add "FIX" as a record option to a theta
498 records. Then when "store_init" was called on the "FIX" option PsN crashed
499 because the option was not a "theta_option". So i added a check for
500 "store_init" abilties... if FIX would become a record option in the future.
502 - Member accessor "datafile" replaced with "datafiles" in model. Affected
503 only the bootstrap which have been updated.
505 - An assumptions about outputfiles crashed sumo occationally. Some more
506 rigorous contentchecking has been added and fixed the problem.
508 Internal changes
509 - Cleanup of dubious utility options.
510 - Option help sorted alphabetically.
511 - "flush", "target" removed from model.
512 - "remove_temp_files" removed from modelfit.
513 - Changed the Makefile considerably, much simpler now.
514 - added "column_to_array" function in data module.
516 Changes from 2.1.9 to 2.1.10
518 Features
519 - The 'picky' option of all utilities now triggers on
520 COVARIANCE STEP ABORTED and
521 PROGRAM TERMINATED BY OBJ
523 Changes from 2.1.8 to 2.1.9
525 Features
526 - Nordugrid support reintroduced (but will probably not be developed further)
527 - Sumo now supports multiple problems and subproblems.
528 - Sumo has new option "precision" which control the number of digits
529 to present estimates with.
530 - An extra 'short' logfile is output from the scm with the most important
531 information.
533 Bugs
534 - Bootstrap option "stratify_on" doesn't work with columns defined in
535 extra data files given with "extra_data" option. Not Fixed.
536 - Resumes of scm's with extra_data_files is now possible. Still some
537 problems with scm's where search_direction=both.
538 - LSF options was not communicated properly between tools and subtools
539 so submission of bootstraps, llp, cdd and scm did not work. Fixed.
540 - Default values for LSF can now be configured in psn.conf.
541 - Print order of modelfiles places $MIX before $PRED. Fixed.
542 - extra_files option was broken. Fixed.
544 Internal Changes
545 - Tool will always fork subtools, this will save memory in the scm (and
546 other recursive tools).
547 - Common options need only be specified once. In one of three categories:
548 model_options, tool_options, script_options.
549 - models can now be created with "reference" objects. This solved the
550 scm resume problem.
551 - Added function to model that cleans extra data code.
553 Changes from 2.1.7 to 2.1.8
555 Features
556 - Rudimentary support for execution on Platform LSF grid systems is now
557 included. The following extra attributes are available to all tools
558 for controlling the execution:
560 run_on_lsf - Use LSF
561 lsf_queue - Use this queue on the grid
562 lsf_options - General options for the grid queuing system.
563 lsf_project_name - Optional. May be used by the grid system
564 administrator to register control project
565 cpu usage.
566 lsf_job_name - Optional. May be used by a grid system
567 administrator to control multiple runs on a
568 grid system.
570 - All tools now have a raw_results file holding the diagnotics and
571 parameter estimates for each NONMEM run.
572 - All tools now handles model files containing one $PROBLEM and no
573 SUBPROBLEMS only.
574 - The Bootstrap and CDD routines now create separate files holding
575 information about which individuals that were included/excluded
576 in/from each created data set.
577 - The Bootstrap results file now has warnings section to better
578 display which runs that were used for result calculations and which
579 were not
580 - PsN now tries to read psn.conf from the users home directory. (Will
581 require some extra documentation).
582 - Optimized modelfit thread operation by removing Storable::Store.
583 - The clean options has four levels of operation.
585 0 - No cleaning at all.
586 1 - Clean more, but make sure resumes work (this is default).
587 2 - Clean more, but make sure summarize work.
588 3 - Clean everything, keeping only results and logs.
590 - Some attributes and accessors of the PsN-Core class 'output' have
591 had their names changed to make it more clear what their purposes
592 are:
593 termination is now called minimization_successful
594 termination_message is now called minimization_message
595 covstep_termination is now called covariance_step_successful
597 these attributes are new:
599 covariance_step_warnings (boolean; 0 or 1)
600 estimate_near_boundary (boolean; 0 or 1)
602 The attribute which was previously called termination could have the
603 values 0, 0.5 and 1 for 'minimization terminates', 'minimization
604 successful but warnings or errors were printed from the covariance
605 step' and finally 'minimization successful'. The new attribute
606 'minimization_successful' can only take the values 0 and 1 and the
607 new attribute covariance_step_warnings now takes the value 1 when
608 covariance step warnings were found. Checks are also perfomed to
609 test whether the estimates are close to any boundary.
610 - Html documents is now generated from command line help text.
611 which ensures that they are consistent and up to date.
612 - Personal configuration file now available under Unix like OS:s
614 Bugs
615 - Options specific to modelfiles were ignored in some Utilties.
616 - Gradients now output in windows. Just like with nmfe.
617 - Bootstrap now consumes considerably less memory.
618 - Removed unused/obsolete "resume" function from modelfit.
619 - Sorted method entries in modelfit diagram.
620 - Fixed bugs in parsing of OFV and eigenvalues.
622 Changes from 2.1.6 to 2.1.7
624 Bugs
625 - "clean" and "compress" options now work under Windows.
626 - SCM now handles crashed runs appropriately.
627 - The ofv_forward, ofv_backward, p_forward and p_backward options of
628 scm now works as they should.
630 Features
631 - SCM improvements. The final model is now available in the
632 first-level scm directory of each step.
633 - The Utilties -help option is improved. You can now give a list of
634 options to -help for specific documentation of those options, like
635 so:
637 $ scm -help do_not_drop
639 - Minor changes in how the [nm_version] section in psn.conf i
640 handled. Look in the psn.conf file that is distributed with PsN for
641 more information.
642 - SDE modelfiles are now supported. (see the -sde_records option)
643 - The update_inits method of the model class now accepts an output
644 file name as input.
645 - Some attributes and accessors of the PsN-Core class 'output' have
646 had their names changed to make it more clear what their purposes
647 are:
648 finalgrad is now called final_gradient
649 parampath is now called parameter_path
650 gradientpath is now called gradient_path
651 covterm is now called covstep_termination
652 - The Utilties now saves the command line that started it in a file
653 "command.txt" under the corresponding directory.
654 - Added a "silent" option to the Utilties that quenches all output.
655 - "nice" option added to utilties that allows you to set priority in
656 unix.
657 - Rudimentary support for NorduGrid reinstated.
658 - Installer script rewritten. It now supports Perl installations in
659 nonstandard directories. Discrepancies between *nix and MSWin32 is
660 smaller.
662 Internal changes
663 - The code that is autogenerated with "dia2code" is optimized a
664 bit and saves a hash "__valid_parameters" in each object, which
665 allows the constructors to be used as a generic (shallow) copy.
667 Misc
668 - continuous spelled correctly throughout SCM.
671 Changes from 2.1.5 to 2.1.6
673 Bugs
674 - Stratify_on bug in bootstrap method fixed.
675 - Data files in directories different from model file are now
676 possible.
677 - Path name handling reimplemented. Relative paths are now handled
678 better.
679 - Models that generate table files would make PsN complain about
680 missing files.
681 - Its now possible to specify multiple output names as a
682 comma-separated list to execute.
683 - The scm can now go backwards.
684 - Handling of table files now responds to the
685 "ignore_missing_output_files" flag.
686 - nmtran error messages was lost in Windows 2K. Execution of nmtran
687 is more plattform independent now. It uses a temporary file, but
688 it is never bigger than the nmtran messages.
689 - Fixed a bug that limited choice of compiler under unix.
690 - Typo in scm utility fixed. (included_relations in backwards step).
692 Features
693 - Added support for "$NONPARAMETRIC" record in model files.
694 - A few improved error messages related to parsing of model files.
695 - The SCM config file now features sections for parameter bounds and
696 user-defined code
698 Internal changes
699 - psn.conf settings are now available as global variables. psn.conf
700 is only parsed once.