Refactor: Split config objects to Gui and Probe

[mentor.git] / TODO

This is a TODO file for Mentor project.  Currently it includes just draft
random ideas about future Mentor development.



--------------------------------------
GENERAL {{{
--------------------------------------

- Design goals:
  open  - set of small utilities (probe, gui, vim) which together form
  something advanced , ability to use open formats : xml, txt , sqlite
  scripting - with python scripting , OLE automation and and KDE KParts
  interface for scripting

  focus on automation/user-friendliness of creating packs
   creation of packs



- General idea [MOVE TO DOCS]
What can be used Mentor for?
Mentor first in supermemo all possible learning means:
cloze deletion
mindmaps
diagrams
uml
conversion from source code
automatic knowledge retrieval

- each config option should have params
  like possible values if stored in settings or constant
  if modifiable from commandline
  if usable in probe or in mentor or in freq

- add to config parse_probe_commandline and parse_mentor_commandline
  ? maybe better will be splitting them ?
  and pass optparse to them
  also will need get_probe_usage and get_mentor_usage

- upgrade the database upon entering
  when opening database should check the result
  if version is lower then ask if proceed with upgrade
   upgrade implemented similarly to upgrade procedure we did use in Vademecum
   Szefa

}}}


--------------------------------------
DOC {{{
--------------------------------------

- Make TODO file more readable
As a first TODO item for this file is make it more managable and readable.
Also convert it to some markup format.

- design the mentor documentation after supermemo and other flash-card program
documentations.


- Check out how python doc is done - how do they generate their documentation??

- Write documentation in LaTeX and then have it exported to other formats:
html
chm
man
texinfo


- doc:
all technical comments: how things are done : should be written directly as
comments in the files
doc-strings should include requirements

}}}

--------------------------------------
DESIGN {{{
--------------------------------------

- make a mindmap
make a mindmap of mentor/probe requirements and design
and gui mockup of mentor
and uml diagrams of mentor to get a general overview first
then go top-down programming -> making stubs for all the procedures and
functions and listing problems to solve --> with comments
solve the problems aside (in special code branch?)
THEN start bottom up -> writing code for the simplest use-cases but in the
described framework


- excel table for class comparison in probe
make excel table with all the class and features they have - how they compare,
how can I generalize them ??!

- code style:
make all Qt upperCaseStyle so that I do not get confused
(all except for the general utils module)

}}}


--------------------------------------
VIM {{{
--------------------------------------

- to get round problem of unescaped chars in highligting :

- dynamic escaping
\command may be ignored if not one of the known commands
 but what about those defined by myself ??

- improve highlighting

- Vim useful functions from mentor
defun mentor-create-item-from-line ()
defun mentor-enclose-region-with (region text)
defun mentor-convert-to-section ()
defun mentor-convert-to-question ()
defun mentor-convert-to-PrbCloze ()
(global-set-key [f6] 'mentor-create-item-from-line)


- vim syntax: subsection in special color ??

- vim syntax folding for subsection in ftpplugin
- read about syntax folding


- dynamically creating highlighting commands
may highlighted only PrbCommands which are really in probe:
and dynamically create them if some macros are used
how it will be possible ?

- maybe double percentage instead of single percentage for comments?
 %% this comments may be in slight different color
 %% this will be double percentage and only on beginning of line
 %% to avoid conflicts
 %% MORE DARKER GRAY ? as I want to discern between text and the rest

}}}


--------------------------------------
GUI {{{
--------------------------------------

- drill window  - asking for selected items


- different views in a docking interface:
  list view : simple and extended
  card view : question + hint, answer + hint
  info view: score, next review, last review reviews average score simple and
  detailed , category? tags?
  source view: with xml or (q and a) view on current item


- import/export from q & a file / xml file (in Mentor format) in SuperMemo
format  in KDE words format ??


- advanced grid view for managing : this should be instead of the tree that is
used in SuperMemo
the grid should be killer feature allowing for very easy access to items
 with the power of excel but easiness of gmail / web interface
 maybe button with select all and action to perform on them
 but on

it should combine the best features on known grids:
excel, plsql dev, sqlnav, tytan, devexpress, lookupedit , koffice spreadsheet,
thunderbird mails view
program :
 in place editing
 sql query on demand
 search text
 filtering by drop down menus
 filtering by advanced queries (not and like)
 sorting on different columns
 in-place editing
 inserting items on specific position (display order)
 multiselection of items
 filtering by specific tag
 ability to show/hide some columns and move columns
 adjustable cell height with auto-wrapping of contents
 right-click menu to run operation on set of elements (e.g. switch to pending ,
 or assign tag/edit tags for the items)
 different color of fields, depending on their status
 selecting the whole row but editing separate cells possibility


- see the spreadsheet, charts etc examples. and model-views examples and code them in python first
to see how a grid example could be implemented
tip: maybe send spreadsheet demo to pyqt-dev to see if they are interested in
the example

- grid columns:
  id question answer qhint ahint example score
   all columns possible in the view

: with sql interface for querying database
 list like excel+plsqldev table for viewing words  with filtering by tag category sorting on different
  filtering like in lookup panel
  ability to hide some columns if necessary

- make it run with debug version of Qt library
 and give it full debug log so that I see what is actually going on !
 see all signals coming out and caught by slots


-  switch to these later when I have function for automatic inserting in
 from clause
 from PyQt4.QtCore import SIGNAL, SLOT
 from PyQt4.QtGui import QWidget, QPushButton, QApplication, QMainWindow, \
      QKeySequence, QAction, QIcon, qApp


- maximally open for extensions:
  add python scripting engine
  in windows add ole automation server ??
   in kde kobject for scripting

- add labels to views : tree view , list view etc.

- tags instead of ategories

- automatically upgrading databases if found from older versions

- checking databases , fixing them as well

- data will be stored in sqlite database each record will be an xml blob plus
parameters on the learning process
  and original file
  e.g. <question> </question>
        <answer>  <hint>yes!</hint></answer>
    or <text>  <cloze> </cloze>  </text>
    should the xml format know about the cloze ?
perhaps it's better to store them in my custom database format and translate it
to xml on demand? or allow the user to edit the xml ?


- what about encoding issues ??
  it should be all utf-8 or ascii ?
  each item separate encoding or the same encoding for the whole database ??


- Mentor stages:
  normal text  --> probe --preformat
  prb text --> vim macros to markup quickly --> probe --export (default) --> qa
  or xml file
  import to supermemo/mentor

- Gui - random key to press  before going to next to prevent mindless swapping
Mentor -
before answer press key with first letter (or two keys)
to disable mindless swapping

- Use word-wrap when displaying items
word-wrap w testing?


- Graphical/rtf version of incremental reading program
 what I do in vim should be possible to be done in gui as well - for people who
 don't have access to vim
 something like incremental reading in supermemo but done in my way

- card view: switching between source (xml view) and render view in tabs
editing xml / rtf but via styles
word editing but with focus on styles so that I can do it also automatically
using xml transformations
but don't forget about novice users: have to somehow combine the
power/extensibility of html with easy editing of the units

- xml template:
<question> </question> <answer> </answer>
 <prefix> </peg> <example>
 <bold> <color 12> <emphasis>

- lazyshow on dialogs exec as well - right now only works for windows

}}}


--------------------------------------
INSTALL {{{
--------------------------------------
- install script NSIS for installation on different windows machines

- script setup.py for building on linux, windows, mac etc
  and building to python

- Building and testing automatically
The build script should also perform a set of tests after making the build
to make sure the build is correct
all the test suite should be run

- move this to build script
 Run pyrcc4 mentor.qrc to mentor_rc.py file
 but only if file mentor_rc.pyc does not exist or is older than mentor.qrc

- add time-sensitive building by default:
by default action build will not touch files like mentor.pdf or mentor_rc.pyc
if they have not been touched after the last build
   but also add option to do a full rebuild

- additional build information:
build info should include version numbers (detailed) of
operation system
Python Qt PyQt libraries
free memory
database used ?
and what other info?? check this out

Building/crash
how to send this info?
Have parameter --systeminfo or button system info in about box
User should paste this information which sho

source sha1 GIT full key this from git
operating system how this one ?? ?
qt version  qt location
pyqt version    ??
python version

In [2]: QLibraryInfo.buildKey()
In [6]: str(QLibraryInfo.buildKey())
In [7]: str(QLibraryInfo.location(QLibraryInfo.PrefixPath))
Out[7]: 'C:\\Programs\\Qt'
In [8]: str(QSysInfo.WinVersion)
Out[9]: "<class 'PyQt4.QtCore.WinVersion'>"
In [10]: print QSysInfo.WinVersion == QSysInfo.WV_2000
False

- problem: build date is not always the same ase git commit date
maybe I should extract the date from the source as well? git show last commit
date and build key from that ?
or maybe embed 7 digits SHA1 instead?


- different building and testing automation
 light testing
  have a test repo and a script which will pull from draft/devel  ,
  rename misc and   other directories and libraries
  build the solution
  run exe tests for both mentor and probe
  and run mentor gui

 more thorough testing :
  have ubuntu  and windowsxp  clean vmware images
   but with needed libraries installed

 they will also pull from my draft repo and build it and test it on clean
   system

}}}

--------------------------------------
PROBE {{{
--------------------------------------

- no question mark at the end of the question


- enable/disable markup easily
  set unset options
  x

- different variants of column questions
in 3 columns
in 2 columns
sep by TABs , sep by many spaces

- how to exclude
some when i column view
% ^\  exclude\{   }   ^{ }
% This excludes the whole item from column view
\begin{verbatim}
aaaa
^\{bbbb
cccc}
ddddd
eeee
^{ffff}

#{here some task }
\end{verbatim}


- remember I have to design more generally (with full commands and all)
 and enable some features as shortcuts and aliases -> see perl where some stuff
 can be done more and less verbosely
 commands may be abbreviated.

- options may be global in preamble and regarding the whole file
  and local - regarding only some subset of file
  e.g.
\begin{verbatim}
\option{swap}
   ...
or \swap{...}
\end{verbatim}

- breaker class -> make it break by comma instead of / character?
  when i don't want comma then i would need to escape it with backslash
\begin{verbatim}
this is, some text, to be more sure
instead of
this is/ some text/ to be more sure
\end{verbatim}

- what shortcuts to use:
\begin{verbatim}
| |  _ _    for cloze \cloze{
< > ^ ^     for set  \set{}
// //  /  /    for breaker \br{}
,, ,,  ,  ,    for breaker2 \br2{}
$ $            for verbatim code
<- ->   TAB    for tabbed breaks
$$
&&
!! !!  hint in answer
?? ?? hin in question
:: ::
;; ;;
<<   >>
\end{verbatim}
but make them work only in selected region so I don't get many false positives.



- note that it's
\begin{verbatim}
\command[]{ }  and not \command{} []
\end{verbatim}

and \\ is used to break line Like \newline
and \ is used to join (but it must be at the end of line - no whitespace after
it!)



- CLASSES:
  set cloze breaked
  are divided by:
-->   what selects from text:
      all, marked, parts, words, sentences, tabbed
-->  when item is broken
       on enter
       on dot sentence
--> what is ignored
--> what selected
--> how breaked , formatted
--> formatted or verbatim , how formatted
--> how to extract
    ----------- -->
       --> -->
--> what to output:
     e.g.. grapical outputs coordinates


- how to find in document my markup quickly
 -> have a special vim function/command to grep all or to jump to next

 -> have special key to unerline \bs pu underline
 -
--> hints
  hint in question \bs\ ? what are they
  hint in answer  \bs\ ! they are what


--> different classes:
  \bs\ sp break on paragraph
  \bs ss break on sequence
  \bs sa break on all



- beginning and ending statements vs. one statements
many statements do not need an ending statement e.g. when \begin
encounters next \begin then it switches to it automatically without any need
for \end - might note in documentation that it is optional
IS IT? this maybe in unsafe  warnings mode
maybe use 2 forms instead : \begin{set} and \end{set}  or \set{     }



- lisp markup - special way to mark hashstrings??  learn them in sm , author
  many documents in them, convert play with them, collect info , read community
  stuff --


- make vim syntax file for the probe language .prb files basing on LaTeX
vim syntax sources.

- all command-line options should be overridable in source -> like verbose mode and debug ??


- using command to a set of words:
\begin{verbatim}
this is \ignore{craze idea} said adam ~{follow me}
\end{verbatim}


- the special chars like \~ should have their counterparts  \ignore

- options should have default values e.g. \lang python  or lang=python



- both forms are possibe either:
\begin{verbatim}
\begin{signature}
\signature
and for ending:
\end{signature}
\endsignature
\end{verbatim}



- what about other chars:
# $ % ^ ' "


- should I use Markdown/Textile language to set formatting in text
  colors emphasises statements ?



- Output template:
should I export it as a question answer or as a xml
this should be in command line options but in the file as well
./probe --output=xml sample.prb
./probe --output=qa sample.prb

\output[template=qa]


- corpus: command line possibility of controling corpus behaviour
 it may use different levels of popularity first 1000 , second 1000 etc.
 does not use any words in first 7000 thousands


- corpus: automatic caching of corpus on start
 automatically cache first 200-300 on enter  and then cache on each read
 also command cache to cache


- corpus: controlling corpus inside file
 \ignored[level=7,cache=2]
 the first 2 thousands should be in cache ??


- ast options improvements:
Parseoption restrictions like in optparse. 14:18 ([[2007.10.11#4]])
Parseoption restrictions like in optparse.


- ast options
also positional and default
options so I can write something like this \title[good] or \title[good, ok,
best] or \title[good, type=ok] options should work as keyword arguments params:
position, name, value


- ast options
exclusive options - some option may not exist at the same time .e.g include ,
exclude - must be able to specify that


- write proper error handling
Proper error handling would require writing a full-blown parser instead of using 00:16 ([[2007.10.24#1]])
quick-regexp
will it make probe much slower ??


- how to discern between parsing and generating
  right now processor class generates all parsed by custom parser classes?
  ParseObject and GenerateObject
  different algorithms used for parsing
  and different algorithms used for generating items.


- IDEA on ParseClassCommand:
put it to code
ParseClassCommand
will be a general class with complex parsing algorithm allowing for many
tweaks by using virtual functions
the tweaks can be used by subclasses
 this will be a complex algorithms
  with many options : markers, switches, other functions
    which may be set by subclass if they want to overrid

  in another case it may select to override the whole parse_content function.

- Formats for output:
testing in formats:
q:
q:
a:
a:

or html like (instead of my . language )
<q>  <a>

<q bimode>
procedure dosomething
<a>



- markups for format:
  \emph \bold  \italic ??
  maybe do it markdown format '' '' for light emphasis  ''' ''' for strong
  emphasis??


- Markups ^ and _
Right now they are superceded by || and // groups
Is is to stay so?
Compromise: too less markup makes the text cluttered
and too much makes more chars needing escaping

Shortcuts aka special chars
There will be a set of special chars which will modify the meaning and
give some clues for extraction purpose.


^   ignored word
this means the word will not be included in repetitions
~  joined words
?   words to be selected

General rule for character ? and ^:
their appearance in text determines whether I should rather exclude or
include words: they are also mutually exclusive - this can be checked as
a test.

Consider these cases:
Poland was raised in ?996.

and
Poland ^was ^raised ^in 996

The first one will ask for 996 only. The second will ask for Poland and
996 as they are not excluded.
By default I ask for all words (unless the \common directive


- General idea about Mentor Probe
Mentor Probe is to SuperMemo incremental reading like Latex is to word 0
Mentor Probe is to SuperMemo incremental reading like Latex is to word
on the other hand it may be possible to develop wysiwyg instead of vim only in
future ??


- use of markup _ ??
 _maybe_  _this would be better_ _than_ _this_ 13:56 ([[2007.10.16#2]])
_maybe_  _this would be better_ _than_ _this_

_maybe this_would_be_better[?what][!clue]
 _than[abbrev] _this
 hinting at the end

- Maybe use lightweight markup to achives formatting for questions?


- corpus: pack/compress corpus tables


- probe import should use other tools antiword, language parsing tools , db
connection tools etc to extract necessary information and present it in a
meaningfull way

 probe import does the general preprocessing vim macros will do the rest and
 last fixes are directly on the mentor database


- Sentence class: no need to escape ? ! outside | | groups
They should be treated as sentence separators .
Of course escaped \? and \! also should be treated as separators.
OR maybe probe --import should escape all of them before ???
BUT sometimes probe --import is not done !


- removing comments before distorts the line number and column number so this
is really a hack

- default class processing
  make it dependant on the mode in which probe was run
  In default mode all is treated as set class
  If in strict mode all needs to be correct?


- Probe Bug:
\code1 class break probe
but not \abrakadabra
I think that's because it begins with \code ??


}}}


--------------------------------------
TESTING {{{
--------------------------------------
But I can try to do the minimum to start learning:
- touch typing rules
- vim program
- vim scripts
- python lang
- python libs
- qt classes and utils

}}}


--------------------------------------
MARKETING {{{
--------------------------------------

- info on site how users can contribute
 filling bug reports, sending feedback and comments
 getting and writing documentation


- publish to sites:
Linux: Freshmeat, KDE, PyQt
Python: python programs
KDE: Planet KDE programs , KDEApps
PyQt : On their Wiki
Wikipedia on learning on supermemo on programs python
Windows software apps: Tucows, Filezilla
Vim: Vim scripts : for Probe language
Learning sites:

- Productivity tips blog
Blog on Vim and Productivity and learning
  and Mentor development on Blogger/Wordpress


- must gather community - see about supermemo and flashcard programs ,
advertise mentor on linux/windows software sites , various distros, python and
qt communities and learning communities
also have databases of different items and advertise them .e.g tell people
there is vim knowledge database for SuperMemo and Mentor
BTW Mentor should be able to import from SuperMemo

- Educational projects promotion:
Kdeedu Edubuntu Eduforge
get in touch with educational project and promote mentor there

- Take part in KDE project (triage bugs, make some easy fixes: to get to know
people and have placed my mentor software) -- KDE 4.0!!

}}}


--------------------------------------
GRAPHICS-MODE {{{
--------------------------------------
- Visio mode supermemo
  graphic counterpart for emacs part
Visio instead of Vi


- 2006-04-18 13:31 folmert: Mentor graphics will have several incarnations:
0. in emacs quick testing (this more like text)
1. visio extended - for quick testing
2. supermemo light - a version of supermemo with ability to test my ideas
and quickly draw schemas and import bitmaps
 and schedule for some time
3. in full supermemo session


- 2006-04-14 13:46 folmert: Mentor Graphics : at best do it as visio extension
the hidden surfaces might be as second layer - may be initially at bottom and
semitransparent



- Mentor.Graphics --

+ full-screen mode
+ bigger image
+ transparent blinks
+ testing capability -- all images and all blinks
only one spot is remembered


Learning Process Menu

Items List - Show For Tomorrow/For After Tomorrow/For Day
Sort By
Filter
all operations on maps showing in List
Show Leeches

Spots Layer
Show spots
Moving spots
automatically generating spots - ability to change them afterward

File - Create, Open, Print, Page settings, Print preview
Edit - Undo, Cut, Copy, Delete
View - Toolbars, Statusbar, Remarks, Diagram, Grids, Rulers, Define rules, Background
Map - Add, New, Delete, Duplicate, (without Goto First/Goto Last?)
Learn - 1. Outstanding, New, Final drill, Test current, Test all, Test selected
Tools - Import -- Probe plugin (seperate app), Copernic plugin
Help - Contents, Index, About



- Export to my graphic supermemo testing ground
MySupermemo

Graphics SuperMemo may be done as a extension to Visio SDK
with objects that will disappear ?? or better bitmap ??
it would be nice to have such access to supermemo that I can export and not
touch it anymore



}}}


--------------------------------------
RESEARCH {{{
--------------------------------------
- other repetitions programs:
 SuperMemo (older versions too), Mnemosyne, FullRecall, Quisition

- other markups:
 RTF, LaTeX, Markdown
LATEX, RTF, Markdown, Wiki, HTML, XML, CSS, SGML learn and author in all possible to
get as many input and ideas ass possible.
Also converting between them etc.

- documentation for other projects
See all big projects and doc how the doc is done professionally: python,
apache, ubuntu docbook etc. , ion uses some special - big opensource projects
what they use , latex resources etc.


- put mentor design in supermemo / mentor itself for better understanding


- research parsing for computer languages and markup languages
TODO see how other parsers (XML, HTML) are written
think about it , design generally then write some specific prototype
Learn LaTeX/TeX well before I begin (using old Mentor first maybe??)


- work with different python and not python parsing engines:
PyParsing, ANTLR and others
try to write my own (HTML, XML, Lisp, Python etc.)

- repetition scheduling algorithm
this will be based on:
Leitner method of repetitions
Artificial intelligence: Neural net, genetic algorithms?  - see mnemosyne ,
SuperMemo published method and Piotr Wozniak notes
common sense and my experience - repeating in increasing periods of time


- repetition parameters/factors
  next repetition period length * factor
  randomness spread * factor
  database difficulty factor
  user skills factor
  how much increase the initial jump:
    +5*1.2   +7*.12

next repetition period lenght factor will be increased or decreased depending
on the notes given , history of learning etc.

perhaps have different algorithms to select next repetition and possibility to
switch between them

- write a simulation package which will show how the item behave in future with
different parameters

}}}


--------------------------------------
HOSTING {{{
--------------------------------------
Google Code Hosting for download/tickets/wiki
Repo.or.cz for Git code repository
Google Groups for mailing list

- ideas about website : how the menu should look like


- website ideas:
Home:
Welcome to Mentor

MENTOR is a tool for improving the learning process.


It can be used as a help in learning foreign languages , geography , science and
can be used everywhere, where there is a need to comprehend knowledge and
somehow memorize it and keep it in memory .


knowledge may be split into seperate items which can be memorized.

By learning process is meant extracting knowledge from written material,
processing it to more consumable form and then memorizing it and storing in
memory.

MENTOR helps by providing ways of extracting knowledge (by summarizing ) and
transforming it (emacs macros) to flashcards

MENU:
Home
News
Downloads
Faqs
Screenshots
Forum
Links
Search
Search personalization

}}}


--------------------------------------
WORKFLOW {{{
--------------------------------------


- have a draft tree for working on many branches on different stuff with quick
commits every now and then
 and move back and forth between branches (pu - feature branches)

- have official tree to which I send patches and which is synchronized with
  other
  the patches are fully described so that I can later get info about the
  project


}}}



# vi:tw=79 fo=qt sw=3 ts=6 fdm=marker