Merge branch 'mob'

[arxana.git] / latex / sbdm4cbpp.tex

%%% sbdm4cbpp.tex -- An advanced hypertext system for Emacs

%% Copyright (C) 2005 Joe Corneli <jcorneli@planetmath.org>

%% Time-stamp: <2006-06-08 14:29:13 jcorneli>

%% This program is free software: you can redistribute it and/or modify
%% it under the terms of the GNU Affero General Public License as published by
%% the Free Software Foundation, either version 3 of the License, or
%% (at your option) any later version.

%% This program is distributed in the hope that it will be useful,
%% but WITHOUT ANY WARRANTY; without even the implied warranty of
%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%% GNU Affero General Public License for more details.

%% You should have received a copy of the GNU Affero General Public License
%% along with this program.  If not, see <http://www.gnu.org/licenses/>.

%%% Commentary:

%%% Code:

\documentclass[12pt]{article}

\usepackage{beton}
\usepackage{concmath}
\usepackage{graphicx}
%\usepackage{hyperref}
\usepackage{makeidx}
\makeindex

%% Reformatting the index to be single column

\makeatletter
\renewcommand{\theindex}{
    \@restonecoltrue\if@twocolumn\@restonecolfalse\fi
    \clearpage
    \section*{Index of Functions}
    \parindent\z@
    \parskip\z@ plus .3pt\relax\let\item\@idxitem
    \addcontentsline{toc}{section}{\numberline {}Index of Functions}
}
\makeatother

%% Run this code to put all of the function definitions into the
%% index.  (I'm not sure yet how to build an index of terms as well as
%% an index of functions.)

%% (while (re-search-forward "\\\\begin{lisp}\n(defun \\([^ ]+\\)")
%%   (goto-char (match-beginning 0))
%%   (insert (concat "\\index{" (match-string 1) "}\n"))
%%   (search-forward "end{lisp}"))

% Use ``makeindex sbdm4cbpp'' to build the index & run latex multiple
% times to get everything settled.

\usepackage{url}
\usepackage{graphicx}
\usepackage[round,authoryear,sort&compress]{natbib}
\bibliographystyle{plainnat}

%% Does this mess up the ``works on a wholly free software system''
%% property?
%%% \usepackage{simplemargins}
%% copying the below from simplemargins.sty, hopefully under fair use:
%% (this will save users from a dependency, whether free or non-free)
 \newlength{\smpagewidth}
 \newlength{\smpageheight}

 \setlength{\smpagewidth}{8.5in}
 \setlength{\smpageheight}{11in}

 \addtolength{\textwidth}{\oddsidemargin}
 \addtolength{\textwidth}{1in}
 \addtolength{\textwidth}{-1in}
 \setlength{\oddsidemargin}{-1in}
 \addtolength{\oddsidemargin}{1in}
 \setlength{\evensidemargin}{\oddsidemargin}

 \setlength{\textwidth}{\smpagewidth}
 \addtolength{\textwidth}{-\oddsidemargin}
 \addtolength{\textwidth}{-1in}
 \addtolength{\textwidth}{-1in}

 \addtolength{\textheight}{\topmargin}
 \addtolength{\textheight}{1in}
 \addtolength{\textheight}{\headheight}
 \addtolength{\textheight}{\headsep}
 \addtolength{\textheight}{-1.5in}
 \setlength{\topmargin}{-1in}
 \addtolength{\topmargin}{-\headheight}
 \addtolength{\topmargin}{-\headsep}
 \addtolength{\topmargin}{1.5in}

 \setlength{\textheight}{\smpageheight}
 \addtolength{\textheight}{-\topmargin}
 \addtolength{\textheight}{-1in}
 \addtolength{\textheight}{-\footskip}
 \addtolength{\textheight}{-.5in}
%% end copied portion.  

%% (And this is what it replaces.)
% \setleftmargin{1in}
% \setrightmargin{1in}
% \settopmargin{1.5in}
% \setbottommargin{.5in}

\usepackage[normalem]{ulem}
\usepackage{amssymb}
\usepackage[mathscr]{euscript}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsopn}
\usepackage{amstext}
\usepackage{amsthm}

\parindent = 1.2em

\theoremstyle{definition}
\newtheorem{prop}{Proposition}
\newtheorem{nota}{Note}[section]
\newtheorem{rem}{Editorial Remark}[section]
\newtheorem{lem}{Lemma}
\newtheorem{thm}{Theorem}
\newtheorem{cor}{Corollary}
\newtheorem{con}{Conjecture}

\usepackage{verbatim}
\makeatletter
\newenvironment{lisp}
  {\let\ORGverbatim@font\verbatim@font
   \def\verbatim@font{\ttfamily\scshape}%
   \verbatim}
  {\endverbatim
  \let\verbatim@font\ORGverbatim@font}
\makeatother

\makeatletter
\newenvironment{idea}
  {\let\ORGverbatim@font\verbatim@font

   \def\verbatim@font{\ttfamily\slshape}%
   \verbatim}
  {\endverbatim
  \let\verbatim@font\ORGverbatim@font}
\makeatother

\newenvironment{notate}[1]
  {\begin{nota}[{\bf {\em #1}}]}%
  {\end{nota}}

\newenvironment{note}[1]
  {\begin{nota}[{\bf {\em #1}}]}%
  {\end{nota}}

\newenvironment{edit}[1]
  {\begin{rem}[{\bf {\em #1}}]\sl}%
  {\normalfont\end{rem}}

\newcommand{\comm}[1]{
\sf \fbox{\begin{minipage}{.75\textwidth}#1\end{minipage}} \normalfont}

\newcommand{\inp}[1]{\sl#1\normalfont}

\newcommand{\xref}[1]{\ref{#1}}

%% Geh! $\chi$ with a subscript is typeset badly if you
%% don't put in a fake superscript, what's up with that?
%% Of course, if you use a function with a fake superscript,
%% you can't very well add another superscipt to it, can you!
%% Grr.
%%
%% But, maybe I could make one required argument and
%% a second optional argument and solve the problem that
%% way? -- Well, LaTeX lets you have one optional argument
%% alright, but it wants the optional argument to be the first
%% argument, and I don't see any easy way around that.  I
%% guess I could ask an expert sometime, but it doesn't quite
%% seem worth it.  OTOH, this issue of typesetting chi's is
%% a bit of a bug, I think, so maybe it should be reported.
\newcommand{\indicator}[1]{\chi_{#1}^{}}

\begin{document}

\title{\emph{Arxana}}

\author{Joseph Corneli\thanks{The content of this file is available
    under your choice of the GNU Free Documentation License (FDL) and
    the GNU General Public License (GPL). If the FDL is selected, be
    advised that this document has no Invariant Sections, no
    Front-Cover Texts, and and no Back-Cover Texts.}}

\date{\today\ (draft)}

\maketitle

%% The system's design draws on ideas from Ted Nelson's ``Project
%% Xanadu'', combining them with the ``semantic network'' strategy
%% from artificial intelligence.

\abstract{A scholia-based document model for commons-based peer
  production informs an advanced Emacs-based hypertext system.}

%% 1. Revise links and backlinks.

%% 2. Put together an appendix with simple specs.  (Done for the
%% basic objects; not necessarily with all of the operations that
%% effect these objects.)

%% 3. After that, there are many user-friendly features that one could
%% tackle.  Additional history lists for multi-dimensional browsing,
%% various useful consistency-maintaining features (not unlike
%% mascons!), versioning support, and so on.  Presumably many of these
%% items can be left for other people to work on after the release.

%% 4. I still haven't completely worked out the theory of
%% "transclusion" (live-updating quotes) to my satisfaction, and this
%% (and its bi-directional variant, "identification") is the "killer
%% ap" I'd like to get into the software before I release it.

\tableofcontents

\newpage

\section*{Preface}

This document may look like a paper, but it is also runable code.
The code implements a hypertext system.  You can use this
hypertext system to browse and edit the code and documentation
that describes the system itself.

To do that, when viewing the document's source code with Emacs,
evaluate this form (in place):
\begin{idea}
(progn 
(require 'cl)
(save-excursion
  (let ((beg (search-forward "\\begin{verbatim}"))
        (end (progn (search-forward "\\end{verbatim}")
                    (match-beginning 0))))
    (eval-region beg end)
    (lit-eval)
  (browse-scholium-system))))
\end{idea}

\section{Prelude} \label{prelude}

\begin{notate}{The philosophy} \label{the-philosophy}
The main concern here is to model systems for collaborative
authorship.  What freedoms and limitations are embodied in such
systems?  Working to understand how to empower users will hopefully
enable us to build a powerful and agile system; indeed, our goals here
extend beyond ``authorship'', to ``readership'' and query-answering;
that is, from hypertext to artificial intelligence.
\end{notate}

\begin{notate}{SBDM and CBPP}
The idea of a \emph{scholia-based document model} (\emph{SBDM}) for
\emph{commons-based peer production} (\emph{CBPP}) is thematic for the
work.  Definitions for these terms suitable to the usage made within
this document are approximately as follows:
\begin{itemize}
\item \emph{CBPP}: the processes by which shared information resources
  are produced by multiple co-authors.
\item \emph{SBDM}: a framework for understanding the ways in which
  documents are put together out of smaller inter-referential
  components.
\end{itemize}
\end{notate}

\begin{notate}{Bits and pieces} \label{bits-and-pieces}
Scholia-based documents tend to be comprised of small pieces whose
semantic content has been illuminated through an ongoing process of
annotation.  This helps with refactoring, and with the process of
producing new, meaningful, derivative works.  Rich annotation also
opens the way to increasingly powerful computer-based inference about
the content of the document.  This document should be a good example.

\begin{figure}
\begin{center}
% \includegraphics[width=5in]{bw}
\end{center}
\caption{Author and summarizing example (\emph{apr\`es} L. Weiner). \label{author-as-scholium}}
\end{figure}
\end{notate}

\section{Introduction} \label{introduction}

\begin{notate}{A scholia-based document model for commons-based peer production} \label{sbdm4cbpp}
The fundamental data structure used in this report is a simple
database made up of \emph{articles}, each of which has several
freeform components.  An article's \inp{name} is an arbitrary LISP
object, typically a string (two articles with the same name are
considered to be the same).  The articles can contain \inp{text} from
a string, buffer, or file (as well as from other sources).  An article
is supplied with \inp{about} and \inp{type} metadata which say,
respectively, what other pieces of the document the articles are
about, and generally how they are to be treated by the system.  A
collection of articles can be viewed as a semantic network, with links
from which \inp{about} data is constituted comprising the edges
between nodes (Note \ref{links}).  (There are other sorts of links in
the system as well, cf. Note \ref{clusions}.)  Linking is how scholia
are attached to other articles; the collection of all articles is
``the commons''; and the rules for interacting with this collection
defines the commons's regulatory system.
\end{notate}

\begin{notate}{Inspirations} \label{inspirations}
One of the first things that got me thinking about this stuff was the
question of how to combine the contents of two digital libraries in a
useful and non-disruptive way\footnote{{\tt
    http://planetx.cc.vt.edu/AsteroidMeta/one\_week\_in\_october}}.
From here, the more technical question of how to do bidirectional
updating between two digital libraries which share content and are
changing in real time arose (see Section \ref{bidi}).  But probably
the most basic motivation comes from thinking about knowledge
representation and human interface for the Hyperreal Dictionary of
Mathematics project.  The aim of this project is to get the wolrd's
mathematical knowledge into a coherent machine-readable and
humanly-useful form.  I anticipate that the first major applications
of Arxana will be those directed towards building the HDM (see Section
\ref{HDM-things}).

After thinking about these things for a while, I wrote up a few pages
that expressed some ideas I later learned to be the central notions in
Ted Nelson's Project Xanadu (see Note \ref{relationship-to-xanadu}).
My interest in AI explains some of the differences between the two
designs; but the fact that I've chosen to use Emacs for development
probably explains even more.

Certainly the powerful text processing tools available in Emacs
(ranging from text properties and overlays to self-modifying code)
make it a great place to develop a next-generation hypertext system.
Existing Emacs-based hypertext systems, including the emacs-wiki
family of projects, and of course the built-in info and help
subsystems, give some sense of how useful Emacs-based hypertext can
be.

Much of the attitude towards AI developed here inherits from Minsky's
``Society of Mind''.  In addition, a certain amount of experience
working with contemporary AI systems (Cyc and KM) both got me to think
about what an AI system is and what I would like to see ``AI work''
become.

The style of writing is inspired by Knuth's ``\TeX: the program''.
Here, instead of a typesetting program that can be used to typeset a
description of itself, we have a hypertext system that can be used to
browse and edit a description of itself.  (Emacs had in fact pulled
that one off previously, and to great effect, but in a rather
different way.)

User freedom is probably the premier philosophical concern underlying
this endeavor.  My ideas about online freedom have something to do
with what I know about MUDs (which isn't a whole lot; mainly that they
are sometimes user-hackable and that they resemble the text-based
single-user games with which I am more familiar) and with my
experience as a participant in various CBPP environments.

Section \ref{philosophy} delves further into these matters.
\end{notate}

\begin{notate}{Straightforward applications} \label{straightforward-applications}
Although I am envisioning the system being used to work on projects of
grand scope and vision, it can also be used in some down-to-earth ways
that are relevant to many people: for instance, the system can be used
by document co-authors to annotate and comment on successive drafts of
an evolving paper.  Features to aid in this process have been present
in MS Word for some time, to the best of my knowledge.  (And in ASCII
you can do it with brackets [ins: like this :ins], OR WITH CAPS.)

(Getting things set up to make quick and easy side-comments on a
given text would be nice.)
\end{notate}

\begin{notate}{Acknowledgements} \label{acknowledgements}
Aaron Krowne both helped inspire the initial design (through a series
of discussions about PlanetMath, the HDM project, and bi-directional
updating between digital libraries) and helped to give it a coherent
initial expression\footnote{\label{corneli-krowne}Joseph A. Corneli
and Aaron P. Krowne, A scholia-based document model for commons-based
peer production. \emph{Proceedings of the Emory University Symposium
on Free Culture and the Digital Library}, 2005.}.

Ray Puzio helped to put the project on a sound theoretical and
philosophical foundation as it developed.
\end{notate} 

\subsection{Overview of the system} \label{overview}

\begin{notate}{Scholia} \label{scholia}
A ``scholium'' (plural ``scholia'') is a comment on something else.
We speak of a \emph{scholia-based document model} (or SBDM for short)
because speech is typically \emph{about} something.  We also refer to
elements of the larger document as ``articles''.  If an article isn't
\inp{about} anything (in the technical sense described in Note
\ref{definition-of-scholium}) then then it is considered to be
degenerate as a scholium.  We will also talk about ``an article and
its attached scholia'' (regardless of whether or not the article is
itself a scholium attached to some other article).  Scholia need not
be pieces of text: they can also be code or markup telling the reader
(or the computer) how to process the article(s) they are written
about.
\end{notate}

\begin{notate}{Generalized links} \label{generalized-links}
The \inp{about} features described in Note \ref{scholia} naturally
associate documents or parts of documents with one another.  This
generalizes the notion of \emph{link} familiar from the web (and other
places).  These familiar links are pieces of text with special markup
which say that are about some other piece of text:
\begin{idea}
<a href="document this link is about">text</a>
\end{idea}
In the scholium system, links can overlap and can point at more
different kinds of targets than in typical hypertext systems.  Links
can also be given rich semantic properties by pairing them with
additional \inp{type} data and appropriately responsive procedures.

It is important to understand the fact that, in our system, markup and
metadata are typically stored as scholia linked to appropriate
targets.
\end{notate}

\begin{notate}{Modeling the commons} \label{modeling-the-commons}
In addition to being capable of modeling many different kinds of
documents, the system developed here should be capable of modeling
and/or supporting the relevant features of the CBPP processes that
create these documents.  The SBDM is actually intended as a sort of
meta-model: a model for modeling models, including models of
documents, commonses, social contracts, culture building, and culture
change.  Cf.  Section \ref{experiments} and Note \ref{the-system}.
\end{notate}

\begin{notate}{Relationship to Xanadu} \label{relationship-to-xanadu}
The system presented here isn't meant to be an implementation of the
Xanadu idea, per se, although it provides some of the features one
would expect from a ``Xanadu implementation.''

I should probably mention that I haven't read all of Nelson's stuff
extra-closely, although I've looked through his books and enjoyed what
I saw there.  As the system here moves towards greater usability, I
assume I'll find the time to review the details from Nelson's writing
more closely and flesh out this comparison.  And I hope to have a
chance to review other systems that follow in the Xanadu lineage, e.g.
{\tt token\_word}\footnote{{\tt
    http://hypertext.sourceforge.net/token\_word/}} (which supposedly
implements ``almost every core feature of Xanadu'').  I guess it's the
hacker in me that makes me want to write the code first; and
certainly, the nature of the code I've produced thus far lends itself
to being helpful with literature review tasks
(cf. Note \ref{literature-review}).  I'm pretty sure that this work
represents an advance, or at least steps in a new direction, for
Emacs-based hypertext.

One clear difference between this system and the Xanadu system is that
here articles are not supposed to be presented in a pay-to-access
fashion.  The idea of working on complicated systems for royalties or
license maintenance doesn't strike me as particularly compelling.
Also, in this system, articles can be deleted (see Note
\ref{deleting-articles}), which as I understand it, is discouraged in
Xanadu (more investigation is warranted and the system for deletion
here needs finishing!).

To sum up, my overall sense is that Arxana, combined with simple
internet protocols could, very nicely, facilitate something
Xanadu-like, suitable for the current age.
\end{notate}

\begin{notate}{When implementing something new}
It is good to have an illustration of why the system is special and
different, and why it is also compatible with existing ways of
working.  This is true whether the system is based on code or mores
(e.g. why are principles of freedom important online?).  I'll be
devoting more time to making these illustrations in the future.  I
expect that such illustrations will do a lot to underscore the value
of the framework developed in this document (and, indeed, value is
often most obvious in the more concrete manifestations and examples of
a more general system).  Cf. Note \ref{principle-of-orthogonality}.
\end{notate}

\begin{notate}{What doesn't work or needs debugging} \label{doesn't}
Here are some more details on things that we're planning to provide
at some point but haven't finished yet.
\begin{itemize}
\item We should support threaded followups and make other improvements
  to the display (see Note \ref{threading}).
\item ``Labels'' work but ``namespaces'' haven't been implemented yet
  (see Note \ref{labels-and-namespaces}).
\item We don't yet have support for ``actionable'' articles (see
  Note \ref{interactive-and-actionable-scholia}).
\item Various browsing features could be improved and extended
  (e.g. to enable deletion or editing of some subset of articles en
  masse), and probably debugged (especially the code from Section
  \ref{temporal}).  The history mechanism could probably be debugged
  by displaying it.
\item We haven't fully finished the code for deleting (Section
\ref{deletion}).
\item Committing could stand to be beefed up with versioning (see Note
  \ref{we-adopt-a-versioned-model}).
\item We don't yet have support for saving articles selectively.  In
  particular, we hope to add support for saving only new or modified
  articles soon (but of course also for other predicates).  
\item Support for derivative articles is somewhat lacking (Section
  \ref{derivative}); in particular, I've done some work and sketching
  for transclusion and identification (Sections \ref{transclusion} and
  \ref{identification}) but I'm not yet happy with the results and
  have instead had to seek a temporary work-around .  These are some
  of the trickest technical problems tackled in this document (see
  Note \ref{difficulties-with-text-properties}).  However, they should
  also be some of the ``killer aps'' when they are finished.
\item Support for working in a distributed environment is lacking.
  I plan to add ``native'' support for CVS and/or GNU Arch
  soon\footnote{{\tt http://www.gnuarch.org/arch/how-arch-works.html}}.
\item Several of the experiments we describe haven't been attempted
  yet.
\item Increased standardization and various other improvements should
  be effected throughout.
\item We need to check and expand support for buffers and files within
  the scholium system; see \ref{principle-of-orthogonality}.  The
  question of scholia attached to the Scholia Display buffer may prove
  to be an especially annoying case; where will these be displayed?
  More generally and less confusingly, what should be done with
  scholia attached to buffers that have frequently-changing content?
  Perhaps these scholia can somehow be associated with those changes,
  and swapped in and out programmatically whenever the changes occur.
  We should debug the `display-scholia-about-current-buffer' function.
  See Note \ref{killed-buffers} for an example of the sort of problem
  we have to deal with here.  (By keeping articles and buffers
  separate, we've made the first step towards insuring orthogonality,
  but we want to be sure of usefulness, too, e.g. by improving default
  loading of scholia for buffers.)
\item We need to check how this loads against a plain-vanilla Emacs,
  and probably provide some easier means of getting started.
\item Make it so that `sch-plain-text' (Note \ref{sch-plain-text}) can
  display links.  In fact, we might want to always have it take in
  links, or the unlistified contents of links.  See Note
  \ref{datatypes}.
 \item We should finish the functions for browsing parents; cf. Note
   \ref{forward-in-parent}.
 \item Has the problem with committing a document that has a links at
   the start of its display been resolved?
\item The system should be documented with use cases for all of its
  features; at least the basic ones.
\end{itemize}
Given that the basic system does work and hopefully provides an
exciting glimpse of what is possible here, now seems to be a
reasonable time to invite other people to help with some of these
issues.
\end{notate}

\begin{notate}{Streamlined versions of this document}
As mentioned in Note \ref{bits-and-pieces}, the system makes it easy
to produce certain kinds of derivative versions of a document.  Two
examples that may be helpful particularly helpful to readers are a
version that focuses on code (see Note \ref{with-code}), and a version
that steers clear of code (see Note \ref{without-code}).  Compare
``weave'' and ``tangle''.  (Of course, I do suggest reading everything
at some point; see Note \ref{motto}.)
\end{notate}

\begin{notate}{A motto} \label{motto}
Certain issues cannot be resolved without writing code.
\end{notate}

\subsection{Design issues} \label{design}

\begin{notate}{Introduction to design issues}
Some of the key \emph{development issues} have been sketched in the
previous section.  Here we discuss the issues that come in at the
level of \emph{design}.
\end{notate}

\subsubsection{Design principles}

\begin{notate}{Principle of minimality} \label{principle-of-minimality}
This system could be infinitely complex.  At each turn, we try to
stick to minimal implementations first and then work out from there.
\end{notate}

\begin{notate}{Principle of order} \label{principle-of-order}
Functions and variables should not be used before they are defined.
(There is a related principle of context that is harder to express:
the idea there is that things should be defined in places that make
sense.)  Any exceptions to this rule should be carefully considered.
\end{notate}

\begin{notate}{Principle of transparency} \label{principle-of-transparency}
It would be nice not to have features that disrupt the easy and
completely general flow of the code, but this seems very hard
to accomplish.  We can at least try to be aware of the places where such
losses of generality appear, and make some assessment of the
consequences (and possible future remedies), and certainly try
to avoid losses of generality whenever possible.
\end{notate}

\begin{notate}{Principle of orthogonality} \label{principle-of-orthogonality}
In order to make this system work well with the rest of Emacs, it
should be roughly orthogonal to the rest of Emacs.  It should not
require you to go off into your own little world, nor should it
require redefining basic functions, or otherwise change what people
are used to in terribly significant ways.
\end{notate}

\subsubsection{System architecture}

\begin{notate}{The principles of the architecture}
The system is built on the principle of a \emph{semantic network}.
Articles correspond roughly to the \emph{nodes} of a graph, and
link-backlink pairs to the \emph{edges}.  Both of these types of
objects can bear labels with certain structured information on them.
(Actually, labels on links are sub-frames of the labels on articles;
and labels on backlinks are derived from data about the linking
article (I guess you could call that an ``induced subframe'' or
``derivative subframe'' -- the issues having to do with where the
knowledge is stored, e.g., where you might store the query that tells
you which thing is a derivative, or one particular sort of derivative,
are in my opinion a philosophical issue beyond the scope of what we
wish to get involved with in this paragraph) and anyway, all
information \emph{in} the system is stored on articles.  But we don't
really need to go there now either.)  We could just say ``frame'' and
``subframe'' and keep it simple.  But note that these are the
\emph{internal} frameworks of the system, not the content frameworks
or whatever you want to call that.  That really just about sums up the
architecture of the system itself.
\end{notate}

\begin{notate}{Metaprinciples of the architecture}
Eventually we want to get things like code that defines the system to
be ``understood'' by the system in some way.  These are semantic
networks, so saying that they can understand something is moderately
appropriate.  Right now, the code is just imported into the system in
a bulk fashion.  But eventually the functions could go into the
article table.  Really, the fact of the matter is that later we want
to put the fundamental object arrays for Emacs into the article table
too, so that the whole computation that is Emacs becomes a massive and
somewhat complicated scholium based document evolving in time.  So,
now we get a sense of what it means to be self-documenting to the
extreme.  (Now, per usual, we just modeling something that is already
there, and is already documented to a certain extent by itself, maybe
to a richer extent then you might think.)  Getting Emacs embedded in
the scholium system is the next step after getting the system somewhat
roughly embedded in itself.  (So, this might be an interesting thing
for the LISP crowd to think about; and indeed, it provides a sort of
metaphor for the structure of the HDM itself.)  

To sum that up, and to conclude! We want to make the system
increasingly cognisant of itself.  If it doesn't make sense to look at
all of the processes going on inside of Emacs, for example, then we
don't look at all of these processes.  But to the extent that it is
useful to look at the details of a given process, it would be helpful
to track that process in the scholium system.  (One question we might
ask ourselves is when exactly is it useful to track a given process,
noting that we are often tracking the process through time, if it is a
computation, and then most likely examining the trace later.  However,
you could also set things up so that you could look at the trace from
different parts of it that are evolving.  Of course, you can't look
into the future, you can only look at nodes that have already been
computed.  But you could get a look-ahead effect going within your
computation that would know where to look ahead of time, so that its
journey to retrieve the answer begins before the answer has actually
been recieved, and the query process meets up with the answer just in
time.  And that would be fairly tricky!
\end{notate}

\begin{notate}{Architectural implementation}
The system is currently implemented as a hash table of articles; the
format for these is explained later on in this document, mainly in
Section \ref{definition-of-scholium}.
\end{notate}

\subsubsection{Stylistic considerations} \label{stylistic-considerations}

\begin{notate}{Backend and frontend}
To the extent that it is possible, we should distinguish between the
backend and the frontend of the system.  We could rewrite the backend
in scheme, for example, and rewrite the frontend in various
web-programming languages to make a system that is easy to use over
the internet.
\end{notate}

\begin{notate}{Abstracting access} \label{abstract-access}
Since we may eventually support both lists and hash tables (and
perhaps other things), we presumably should have an abstract means of
accessing articles.  (Unless we were to decide to just do things with
hash tables; however, it seems likely to me that even if we use a hash
table as the main library, people may want to use lists for
sublibraries; if the same functions for access are supposed to be used
in both full and sub library scenarios, we'll want the abstract access
regardless of how the full library is set up.)  A theoretical
discussion of the benefits of hash tables can be found on
PlanetMath\footnote{{\tt http://planetmath.org/encyclopedia/Hashing.html}}.
\end{notate}

\begin{notate}{The Emacs Way} \label{the-emacs-way}
Scholia about files and about buffers are probably most of what would
make this system useful to people who are collaborating on papers or
coding projects.  See Note \ref{bias-towards-buffers}, Note,
\ref{back-to-base}.
\end{notate}

\begin{notate}{Intuitive interface}
The bread and butter of the system's user interface is a set-up with
side by side buffers (see Note \ref{side-by-side-buffers}).  Other
display styles (e.g. as described in Note \ref{references}, or
threading, as in Note \ref{threading}, or a display spread across
several frames, etc.) should be straightforward to add.
\end{notate}

\begin{notate}{Text properties and overlays} \label{text-properties-and-overlays}
We essentially need to have scholia encoded locally as text
properties, because we need to be able to cut and paste the elements
and have the metadata transfered along.  However, at least one aspect
of text properties is already ``owned'' by fontlock, namely faces.  To
use faces in the scholium system, we have to hack an override in --
namely, we mark the buffer up using text properties and then mark them
up again with overlays.  In general, text properties are used to track
text, and overlays are just used as a visual aide.  See section
\ref{marking-things-up}, also Note \ref{reparsing-after-editing}.
\end{notate}

\begin{notate}{Managing text properties in buffers} \label{managing-properties-in-buffers}
We store information about \emph{all} attached scholia on text
property lists.  We should also be storing the name of the article on
all of the text throughout the buffer, so that if the contents of that
buffer are cut and pasted, the text can be tracked.  This requires a
somewhat complex data structure, and makes markup with overlays (as
well as some other details, e.g. text marked up with references which
is subsequently cut in half) tricky either in practice or
conceptually.  See Note \ref{reparsing-after-editing}.
%% (These issues are rather different from the topic of managing
%% properties associated with articles \emph{themselves}, which is
%% discussed in Section \ref{managing-properties}.)
\end{notate}

\begin{notate}{Selection methods} \label{selection-methods}
Various selection facilities need to be implemented.  A little
predicate matching thing would make things extensible!  (See also Note
\ref{selective-displays} and Note
\ref{predicate-support-for-listing}.)
\end{notate}

\begin{notate}{Local navigation}
It is important to have convenient ways to move around from scholium
to marked region and back, from marked region to marked region, and so
on.  See Section \ref{local}.
\end{notate}

\begin{notate}{Color and style of display} \label{color-and-style}
We could offer the user a choice between \inp{type}-based coloration
(indicating what sort of scholium is being marked up),
owner-coloration (who created the scholium) and connected-coloration
(which what I've initially coded up).  We could use some combination
of color and other visual indicators (see Note \ref{monochrome}) to
provide more information.

Similarly, text written by a given author in a shared file could have
a special color.  Another type of display would indicate how many
scholia had been created about a given region (like a rainfall chart).
Different styles can be generated essentially indefinitely.  Another
user-friendly feature would be to be able to easily change the color
associated with some particular set of data (rather than changing the
overall scheme whereby colors are associated with data).
\end{notate}

\begin{notate}{The monochrome lifestyle} \label{monochrome}
For the benefit of people working with and without fontlock, three
display modes to point to attached scholia from Window $A$ will be
available: one that uses font lock; one that uses intangible tokens
{\tt [N:]} and {\tt [:N]} where {\tt N} is a number to demarcate the
beginning and end of the $N$th scholium (or some other customizable
pattern, e.g. based on the bookkeeping information); and a third
``relaxed'' mode uses no markup at all.  A number of intuitive ideas
for inserting indicators (e.g. in the fringe) could be tried.  See
\ref{blinking} for another monochrome-friendly approach.  Another
simple idea, similar to the one described above, would be to use
% Thanks /sw/share/texmf-dist/source/plain/impatient/
symbols (e.g. $*$, $\#$, \S) to associate scholia with the
beginning(s) of the passage(s) they are attached to, and use narrowing
to screen out other material.
\end{notate}

\begin{notate}{User-crafted semantics} \label{user-crafted-semantics}
As a generalization of Note \ref{color-and-style}, we will eventually
want some essentially completely different styles of rendering.  In
every case we should also move to support \emph{user-crafted
  semantics}.  See also Note \ref{types-of-text}.
\end{notate}

\begin{notate}{Contextual display} \label{contextual-display}
An interesting feature would be to restrict to showing only the
scholia that are attached to the current portion of the main article
that is currently displayed, or only those associated with the
character position of point.  In order to pull this off, we will need
to look some into page and cursor motion in Emacs\footnote{Thanks to Stephen Monnier and Eli Zaretskii for
telling me what to look for, on the emacs-devel list, 2006-04-04.  The relevant tips were to look at:
`window-scroll-functions' (when a window's starting pos changes);
`fontification-functions' (when a piece of text needs to be displayed);
`before-' and `after-change-functions' (when a piece of text is modified);
`window-start' and `window-end';
`pre-' and `post-command-hook' (to run before/after each command, so you can
detect cursor motion).}.

Alternatively, we could write a new motion function, something like
`move-to-next-region-with-scholium' (cf. Note
\ref{move-to-next-or-previous-region-with-scholium}), and combine this
with a selective scholia display that will show whatever scholia are
found at the new point.  The closest thing I could find in Emacs is
`calendar-move-hook'.

Yet another thought along these lines would be to have scholia appear
in the same buffer as the rendered main article, but, for example, in
columns beyond column 72 (or whatever the rightmost character in the
display of the main article is).

Compare Note \ref{displaying-nearby-scholia} for thoughts on automatic
\emph{contextualization} of an article at display time.
\end{notate}

\begin{notate}{References} \label{references}
Traditional hypertext links are a useful specialization of the general
sort of links we're developing here (Note \ref{generalized-links}).
In an effort to reduce confusion, when we wish to be precise, we will
refer to the scholia that are used to make a piece of text point at
other piece of text as \emph{references}.  This usage is consonant
with use in other hypertext systems, in which ``links'' are pieces of
text that are marked up with references to other texts.

Despite the time it may take to get used to the name, the familiar
\emph{feel} of references should make it easy for people familiar with
other hypertext systems to transition to this system.

References are an example of a type of scholium that appears as markup
instead of appearing ``alongside'' the article they are attached to
(which is what will typically happen with other scholia).  References
may have special semantics associated with them, see Note
\ref{special-reference-types} and Note \ref{fancy-references}.

It may be convenient to offer allow switching between our overlayed
references and the more familiar inline, wiki-like, references, or at
the very least, some other way to easily edit reference targets (and
similarly for other sorts of links).  See Section \ref{wiki} for
various information on simulating a wiki with the scholium system.
\end{notate}

\begin{notate}{Threading} \label{threading}
We can also provide facilities for showing all attached scholia
together with \emph{their} attachments, and so on, within some fixed
graph distance, beyond which we just include links to lower levels (as
yet \emph{unimplemented} - but a good idea).  This is essentially
what's done on Slashdot, and we would want to use a similar indenting
(and, ergo, threading) mechanism.  (We could be even more
Slashdot-like and do scoring and the whole bit.)  The thing to bear in
mind is that we dont' have to be constrained to follow threads along
one ``follow-up-to'' axis; we can choose arbitrary conditions to
constrain the way we view a local hierarchy (cf. Note
\ref{hierarchy}).  The simplest one to state is the one already
mentioned, namely, viewing all attached scholia to some depth.
\end{notate}

\begin{notate}{Ancestors} \label{ancestors}
We typically think about showing a article together with articles that
are about it (its scholia) but we can just as well show it together
with articles that it is about (its referents).  Indeed, views that
take into consideration the article's parents and grandparents have
obvious applications, e.g., when attempting to follow a conversation.
Particularly if an article has no children itself, it may be handiest
to display it together with its parent (multiple parents could
potentially complicate the matter).  These issues have been taken up
to some degree in various places in this document, including
Note \ref{more-than-one-parent} and Note 
\ref{add-visible-parent-and-sibling-links}.
\end{notate}

\begin{notate}{Expand and collapse backlink context}
We typically show an article together with the text of its scholia,
but we could optionally simply say that such-and-such an article links
here.  Compare Note \ref{contextual-display} and Note \ref{ancestors}.

We may also want to provide a hierarchy of additional scholia, mapping
along the links-to and the linked-to directions, see Note
\ref{threading}.
\end{notate}

\begin{notate}{Labels and namespaces} \label{labels-and-namespaces}
A \emph{label} is a predicate extent that has (typically) been
maintained on the fly (or otherwise made concrete).  A
\emph{namespace} is a place to put articles; articles in different
namespaces can have the same names but not be the same.  An object can
have many labels but can only live in one namespace.  (See Section
\ref{subcollections}, and Note \ref{cost-of-maintaining-labels} in
particular.)
\end{notate}

\begin{notate}{Filtering} \label{filtering}
Filtering is closely related to the topic of subcollections mentioned
in Note \ref{labels-and-namespaces} (and it also relates to the issue
of predicate mapping; see Note \ref{mapping-across-a-subcollection}
and Note \ref{search-engine-interface}).  The point to keep in mind
is that different people are going to want to see different things
when looking at a document.
\end{notate}

\begin{notate}{Transcluding and outline views} \label{transcluding-and-outlines}
Note that instead of text stating where the URL is, we could have an
``transclude'' scholium; this would cause the latest, current, version
of the target document to appear in the current document when the
latter is rendered.  One should be able to look at the current
document as an outline (hierarchy) showing these ``transclusion''
relationships explicitly.  Similarly, one should be able to collapse
and expand different parts of the document (i.e., in the collapsed
view, only the title of the transcluded document would appear, but in
the expanded view all of the content would appear).  These sorts of
ideas are handy whenever thinking about container documents (Note
\ref{container}; oppose Note \ref{thread}).  When examining an
outline, one should be able to jump to the article in its place inside
of the document that is being outlined, or as a stand-alone item.
\end{notate}

\begin{notate}{Rendered and source views} \label{rendered-and-source-views}
A more mainstream example than the expanded and collapsed views from
Note \ref{transcluding-and-outlines} are rendered and source views.
These could be relevant when working with \LaTeX\ or HTML (e.g., when
browsing this document, it would be nice to have things detexed).
Presumably edits would take place in the source view (though not
necessarily; compare the Emacs Muse\footnote{{\tt
    http://www.mwolson.org/projects/MuseMode.html}}).  Switching in
and out of rendered views would be an extra step when initiating edits
(see Section \ref{initiating-edits}).

One simple feature that would be a nice part of a reasonable rendering
would be to keep the text within a certain width by filling at render
time (this would be both for articles and for the scholia display).
Presumably exported text would also have to be filled if it is to
look good.  

Much more generally, it would be nice to have a full-fledged
\LaTeX\ renderer (along the lines of \LaTeX-to-HTML).  A renderer is
in many ways similar to a parser, so a simple parser is a reasonable
first step.  Some of the work related to 
Emacspeak\footnote{{\tt http://emacspeak.sourceforge.net/}} is,
I think, relevant to this problem.
\end{notate}

\begin{notate}{Page numbers}
If we could figure out how to do some \emph{truly} device-independent
rendering of \LaTeX\ documents, one particular thing we'd presumably
have come up with be a correspondence between \emph{page numbers} in
the printed version of a document and in the electronic version.  In
the mean time, maybe we can come up with some easy way to solve this
using some specially logging when \LaTeX\ processes ``notate''
environments?  (E.g., it wouldn't be hard to use special commands to
make an index of the notate environments in the document; and
presumably Emacs could read the source of that index.)
\end{notate}

\begin{notate}{Alternate names for buffers}
Should buffers have special names stored in buffer-local variables, or
should we search the library for buffers and check to see if they are
the same (regardless of what name they go by)?  Or some other approach
to deal with named buffers?  See section \ref{display-interface}.
\end{notate}

\begin{notate}{Saving and reading}
The user can save and read back a given collection of scholia.  As an
example, arbitrary files can be opened in `scholia-mode' so that all
scholia stored in the same directory as the file will appear
automatically as annotations.
\end{notate}

\begin{notate}{Versioning} \label{versioning}
In order to edit and stay kosher, we should provide support for
versioning.  This version information will be stored on the
\inp{bookkeeping} slot (or, alternatively, we should store sufficient
information to recover the versions; if such information isn't already
available elsewhere in the system).  Note that we could just go ahead
and make functions for editing the text and then add functions for
saving the previous version later, in a hook.  (This relates to the
slightly more general problem of making editing work in general, see
Section \ref{editing}.)

In order to deal with \inp{about} data that terminates on a given
article version instead of on the latest version, we will want to have
an additional optional column available in links that is reserved for
the version number.

(Note that some objects may not need to have versions stored about
them.)
\end{notate}

\begin{notate}{CVS/Arch/filesytem support} \label{vc-and-filesystem-support}
We need to figure out how to integrate external information resources
into the system.  One issue with CVS and Arch is that we (sometimes)
will want to only look at files that are new or have changed upon
reloading.  This means that the version management software will
probably have to be integrated fairly closely into the system.
Another option would be to keep a record of the files in any directory
that is being used, and check this record against that file's listing
whenever the user wants to reload the content from that file.  This
may be the only option we really have for using the filesystem
directly, but for the case of CVS, proper integration would probably
be better.

Note that Arch has its own way of doing version management (as do
all version control systems, of course).  It might be best if
we could swap different methods for version control in and out.
Fundamentally, we need to know where to look for articles --
they don't necessarily have to be in the filesystem (perhaps
they could be assembled in real time in some buffer, for example;
similarly, I think Arch builds various patched versions in real
time).
\end{notate}

\begin{notate}{Responsiveness to editing} \label{responsiveness-to-editing}
As editing takes place, the information in the system needs to be kept
up-to-date and coherent with itself.  The main developments in this
section are presented in Section \ref{committing}.
\end{notate}

\begin{notate}{Unique identifiers for individual pieces of markup}
When committing edits, we reparse the buffer to find the places where
the scholia are attached (Section \ref{parsing}).  For things to
work properly, each piece of markup should have a unique identifier
(the article name and link number should suffice, see Note
\ref{format-of-about-field}).  This will make it possible tell whether
a piece of markup has been completely deleted, or swapped with another
piece, or split into several pieces, or whatever (in theory, a region
can even be moved between buffers, which would require an additional
level of reparsing, we won't worry about that here for now; see Note
\ref{aggressive-dop}).
\end{notate}

\begin{notate}{Manipulation of windows}
The somewhat aggressive manipulation of windows used in this program
could get annoying, especially in the case of editing multiple
articles at once.  (Which is one possible reason for editing things
inside one buffer, see Note \ref{identification-use-case}.)
\end{notate}

\begin{notate}{Sorting scholia}
We want scholia to appear in some reasonable order relative to the
order of marked regions.  Section \ref{sorting} is supposed to do
this.  Note also that sorting should not be destructive; see Note
\ref{order-of-about}.
\end{notate}

\begin{notate}{Complexity issues}
Some features simply would take too long to run when there are lots of
scholia or articles involved.  We need to be aware of these sorts of
complexity issues.  (It usually doesn't take much more than a simple
test to make one fully aware of the problem, when dealing with
terrible complexity scenarios!)  Section \ref{subcollections} contains
a few thoughts on how to limit complexity.
\end{notate}

\begin{notate}{Ease of use}
In addition to being more featureful than the wiki medium, I hope that
this system will be easier to use -- wikis as I know them are fairly
laborious; lots of steps are needed to edit and save.  The system
should be designed in a way that requires few if any ``extra'' steps
from authors.
\end{notate}

\begin{notate}{Return to base representation} \label{back-to-base}
It might be nice to have a way to return from a scholium to its
``actual'' base representation, i.e., if it is associated with a
buffer, we should find that buffer, or if it is associated with a file
we switch to a buffer visiting that file.  Indeed, I think we have
thing set up now so that an article stored in a buffer will always be
attached to that buffer.  We might want to be more flexible, allowing
various different ways to attach and reattach (compare Note
\ref{movable-scholia}).
\end{notate}

\begin{notate}{Code transparency}
It is good to have easy-to-understand names for functions that will
appear over and over again; for example, we provide functions to
access the bits and pieces of scholia (see Note
\ref{access-functions}).  But we should try to do this all the time,
for all of the data structures we build.

See also Note \ref{assimilating-data} for further thoughts on why it
is a good idea to do things this way.
\end{notate}

\begin{notate}{Creating a scholium about several articles}
I'm not sure we have sufficiently strong mechanism for making scholia
that apply to several other articles.  That would be one potential use
of the list browser from the previous section; we could mark articles
and make a scholium about marked articles quickly, or mark articles
and then make scholia about sections from other articles.  Also, it
would be good to have a way to break out of the region selection
routine and start selecting regions from some article fluidly when
working on making scholia \emph{that} way.

This is useful as long as we are able to gather all of the articles we
want to make a scholium about into one listing, or if we can find a
way to use several listings together intelligently.
\end{notate}

\begin{notate}{Space versus time tradeoffs}
We value time over space tradeoffs.  If the data set becomes very
large, this judgement may have to change (e.g. if the system gets
popular, we won't be able to hold all of the available articles
in memory, or on disk, and we'll have to get clever about
swapping things around).
\end{notate}

\begin{notate}{Support for systems in which passages have special meanings} \label{systems-with-special-passages}
RMAIL, info, and EDB are just a few examples of Emacs subsystems whose
most ``meaningful'' objects are smaller than the files in which these
objects live.  Support for these objects would be good to add to the
scholium system, so that we can e.g. record mail messages or info
pages that match some particular property.
\end{notate}

\begin{notate}{Undoable editing}
It would be nice to be able to undo the editing actions that
we support.
\end{notate}

\begin{notate}{Support for narrowing}
Frequently the function `point' is used here; I'm assuming that
documents will be widened properly.  But it would be good to be able
to support narrowing (see Note \ref{systems-with-special-passages}).
\end{notate}

\begin{notate}{Hooks and `cond' forms} \label{transforming-cond}
At some earlier point, it seemed useful to think about transforming a
`cond' form with an unquoted variable that contains extra cases (that
are to be defined as needed).  This seemed like a good opportunity to
use LISP self-modifiability, and furthemore, it seemed like a good way
write things in order (see Note \ref{principle-of-order}).  However,
typically the same thing can be achieved with hooks.  The function
`sch-plain-text' is a good example of this (Note
\ref{sch-plain-text}).
\end{notate}

\begin{notate}{Multidimensional visual browsing} \label{multidimensional-visual-browsing}
It might be handy to have a small control pannel at the bottom of the
main article display that could be used to go backwards and forwards
according to the layout of the document \emph{and also} according to
the order in which things had been browsed so far (and anything else
that the user might dream up to put there).
\end{notate}

\begin{notate}{Using datatypes} \label{datatypes}
In order to be able to pass different kinds of objects around and have
them interpreted correctly, it might be advantageous to have our data
structures typed; I'm not quite sure how types work in Emacs -- but I
can say that datatypes are obviously scholia.  For example, we might
want there to be an article datatype (right now, there are only types
\emph{of} articles, like ``references'' for example).  Another entity
that could have a special datatype is the link.

The question of datatypes for \emph{forms} seems interesting; it
basically boils down to the question as to whether we add scholia to
arbitrary lisp structures (see Note \ref{functional}).
\end{notate}

\begin{notate}{Hooks as scholia}
Hooks are an easy-to-see example of scholia in the Lisp paradigm
(compare Note \ref{functional},
\ref{including-transcluding-and-lisp}).
\end{notate}

\subsection{Elisp requirements}

\begin{notate}{Packages}
I'm not sure we actually need align any more, but cl is
of course very helpful.
\end{notate}

\begin{lisp}
(require 'align)
(require 'cl)
(load "cl-seq")
(load "cl-extra")
\end{lisp}

\subsection{Lisp preliminaries}

\begin{notate}{On `add-or-replace'}
It seems that this function is no longer used.
\end{notate}

\begin{lisp}
(defun add-or-replace (lis elt pred)
  (let ((found nil)
        (n 0)
        (len (length lis)))
    (while (and (not found)
                (< n len))
      (when (funcall pred (nth n lis))
        (setcar (nthcdr n lis) elt)
        (setq found t))
      (setq n (1+ n)))
    (if found
        (1- n)
      (nconc lis (list elt))
      n)))
\end{lisp}

\begin{notate}{On `add-to-or-start-list'} \label{add-to-or-start-list}
This is is used to start store an \inp{element} on a
``\inp{possible-list}''; unlike `add-to-list', it works even when list
is nil, or even (for now) when \inp{possible-list} is not a list at
all.  (Unlike with `add-to-list', \inp{possible-list} should not be
the name of a list.)

This function is used by `put-type-labels', `put-backlinks' and
`label-article'.  Note that you have to \emph{save} the result if you
want to this function to have a lasting effect.

Like `add-to-list', this function will only store \emph{one} copy of
\inp{element}.
\end{notate}

\begin{lisp}
(defun add-to-or-start-list (possible-list element)
  (cond ((null possible-list)
         (list element))
        ((listp possible-list)
         (if (member element possible-list)
             possible-list
           (append possible-list (list element))))
        (t (list possible-list element))))
\end{lisp}

\begin{notate}{On `next-single-property-change+predicate'}
Like `next-single-property-change' except instead of automatically
judging changes in the text property by the `eq' predicate, it allows
the user to supply a \inp{test}.
\end{notate}

\begin{lisp}
(defun next-single-property-change+predicate (pos prop &optional test)
  (let ((starting (get-text-property pos prop))
        (index pos)
        (cmpfct (or test
                    'eq)))
    (save-excursion
      (while (and (setq index (1+ index))
                  (< index (point-max))
                  (funcall cmpfct
                           starting
                           (get-text-property index prop)))))
    (when (< index (point-max))
      index)))
\end{lisp}

\begin{notate}{On `delete-all-dups'}
This function differs from `delete-dups' in being
non-destructive and in keeping none of several `equal'
occurrences of an element in \inp{list} as opposed to one;
otherwise, it manifests a similar idea.
\end{notate}

\begin{lisp}
(defun delete-all-dups (list)
  (let ((tail list)
        ret)
    (while tail
      (cond ((member (car tail) (cdr tail))
             (setq tail (delete (car tail) tail)))
            (t
             (setq ret (cons (car tail) ret)
                   tail (cdr tail)))))
    ret))

(defun set-difference (A B)
  (delete-all-dups (append A B)))

(defun zip (A B)
  (let ((ret (make-hash-table)))
    (while A
      (puthash (car A)
               (car B)
               ret)
      (setq A (cdr A)
            B (cdr B)))
    ret))

(defun flatten (list-structure)
  (apply 'concatenate 'list (list list-structure)))
\end{lisp}

\section{Scholia-based documents} \label{sbdm}

\subsection{The digital library} \label{digital-library}

\begin{notate}{On `article-table'} \label{article-table}
We begin with an empty library.
\end{notate}

\begin{lisp}
(defvar article-table
  (make-hash-table :test 'equal))
\end{lisp}

\begin{notate}{Structure of articles} \label{structure-of-articles}
Every article has a \inp{name}, which can be any LISP object (but is
most frequently a string).  They also have various other standard
sorts of data associated with them; see Note
\ref{definition-of-scholium} for the particulars.  This data is
referred to by `put-article' collectively as the article's \inp{value}
(but this name is somewhat misleading, because the \inp{name} too is
an important part of the article).  One special sort of article is a
``library subcollection'' -- see Section \ref{subcollections} for more
information on these things.
\end{notate}

\begin{notate}{On `put-article'} \label{put-article}
This destructively adjusts the values on the hash table (see
documentation for `puthash').  To \emph{intelligently manage} the
values in the table takes more work, so we typically `get-article'
before to make various comparions and adjustments to existing values
before making changes.
\end{notate}

\begin{lisp}
(defun put-article (name value)
  (puthash name value article-table))
\end{lisp}

\begin{notate}{On `get-article'}
Basically just `gethash', but since the article table is indexed by
\inp{name} and we don't want to pass \inp{name} around all the time,
we add it to the recalled value.
\end{notate}

\begin{lisp}
(defun get-article (name)
  (let ((content (gethash name article-table)))
    (when content
      (cons name content))))
\end{lisp}

\begin{notate}{On `name-of-current-article'} \label{name-of-current-article}
The ``current article'' is of special importance for display (an
article is made ``current'' when `display-article' displays it).  The
name of this article is stored in this variable. 
\end{notate}

\begin{lisp}
(defvar name-of-current-article nil)

(defun current-article ()
  (get-article name-of-current-article))
\end{lisp}

\begin{notate}{What about using the file system?}
If the library is being updated by many people at different times, it
may also make sense to read documents from a file system on the fly.
Indeed, the program does support files and buffers.  But the article
table is still a decent \emph{model} -- similar in nature to a
library's \emph{catalog}.
\end{notate}

\subsection{Creating scholia} \label{scholium-function}

\begin{notate}{Component pieces of scholia} \label{definition-of-scholium}
The `scholium' function maintains a catalog of articles, indexed by
name (Section \ref{digital-library}).  Each article contains
\inp{text}, an indication of what the article is \inp{about}, its
\inp{type}, and \inp{bookkeeping} information which gives the
article's edit history and ownership information.  Articles must
always follow this pattern.
\end{notate}

\begin{notate}{Format of `name' field}
Any LISP object can be the \inp{name} of a scholium.  Scholia are
indexed by name: names are unique within namespaces (see section
\ref{subcollections}), and the combination of name and namespace is
how an article is looked up.  (At present, names are typically
rendered as strings without any newlines, but this is a front-end
matter and essentially arbitrary from the point of view of internal
format.)
\end{notate}

\begin{notate}{Format of `text' field}
Just about anything can go into the \inp{text} field, but anything
``weird'' needs to be backed up by appropriate \inp{type} metadata
and appropriate handlers (see Section \ref{rendering}).
\end{notate}

\begin{notate}{Format of `about' field} \label{format-of-about-field}
The \inp{about} field is a list of \emph{links} (in the sense of Note
\ref{generalized-links}).  See Note \ref{links} for details.

We could at some point investigate making \inp{about} data (and
perhaps other sorts of data) \emph{typed} so that we can more easily
the genesis of a given datum.
\end{notate}

\begin{notate}{Links} \label{links}
Each link is associated with precisely one article.  Links can have
additional link-type information stored about them.  The general
format of a link is:
\begin{idea}
(<target article's name> &rest <link-types>)
\end{idea}
The design here maintains partial symmetry between the treatment of
article types and link types; the cdr of a link can be processed
by the same typedata processing functions as the \inp{type}
data from articles.

For example, the link
\begin{idea}
'(foo (passage 10 15) mistaken)
\end{idea}
indicates that the region from character position 10 to 15 in the
article named ``foo'' is ``mistaken'', whereas the link
\begin{idea}
'(foo mistaken)
\end{idea}
indicates simply that the article \emph{foo} is mistaken.
\end{notate}

\begin{notate}{Links indicate at most one passage} \label{links-indicating-passages}
It seems reasonable to me to assert that each link will indicate at
most \emph{one} passage.  This is indeed an assumption made in the
code, see the function `typedata-includes-passage' (Note
\ref{typedata-includes-passage})).
\end{notate}

\begin{notate}{Format of `type' field}
The \inp{type} field is supposed to be a symbol or list of symbols.
If it is nil, the article is assumed to be normal text.
\end{notate}

\begin{notate}{Format of `bookkeeping' field}
The \inp{bookkeeping} field has a special format, very similar
to the format of metadata articles (see Section 
\ref{metadata-articles}); one might expect to see something like
\begin{idea}
((owner <owner>)
 (ACL <ACL>)
 ...)
\end{idea}
Remember that the \inp{bookkeeping} is \emph{owned}, so we probably
shouldn't put anything there that can't be edited by the user, and so
in particular, things that are needed for system consistency should go
into metadata articles instead.

Do notice that various schemes for access and ownership can be
considered.  (E.g. maybe anyone on the \emph{access control list}
(ACL) can edit the text of the article, but only the owner is allowed
to edit the ACL itself.)

It would be worth looking at existing strategies (e.g. as found on
PlanetMath) for handling complicated ownership and access
arrangements.  Actual implementation of useful bookkeeping features
will have to come later (for a sort of silly placeholder for the
meantime, see Note \ref{sch-book}).
\end{notate}

\begin{notate}{Relationship of bookkeeping data and metadata article}
Bookkeeping data is \emph{user editable}, but metadata article
generally isn't, or, when it is, it is typically treated as common
property.  This is important!  See Note
\ref{metadata-article-overview}.
\end{notate}

\begin{notate}{Access functions} \label{access-functions}
A few simple functions to get pieces of (just aliases to help with
coding; see Note \ref{principle-of-transparency}).  Note
\ref{link-accessors} talks about something similar for links.
\end{notate}

\begin{lisp}
(defalias 'scholium-name 'first)
(defalias 'scholium-text 'second)
(defalias 'scholium-about 'third)
(defalias 'scholium-type 'fourth)
(defalias 'scholium-bookkeeping 'fifth)
\end{lisp}

\begin{notate}{Order and interpretation of `about' field} \label{order-of-about}
The order of links in the \inp{about} field is not arbitrary.

The order of links determines link-ids, which are used for
both  backlinks (Section \ref{backlinks-section}) and markup
(Section \ref{marking-things-up}).
\end{notate}

\begin{notate}{Link accessors} \label{link-accessors}
Links have exactly one linked-to-article together with any number of
different link-types.
\end{notate}

\begin{lisp}
(defalias 'linked-to-article 'first)
(defalias 'link-type 'cdr)
\end{lisp}

\begin{notate}{Special link accessors}
First we need a function that will give us the type-element associated
with a given type.  Then, we have some important examples.

The function `link-type-accessor' will produce the (first) specific
type-element from \inp{link} that matches the given \inp{type}, if one
exists.

Note that the format of \emph{version} access should follow the
`link-version' function given here.  (Passages don't have versions, so
it doesn't make sense to put a version number as an additional element
of a ``passage'' element.)
\end{notate}

\begin{lisp}
(defun link-type-accessor (link type)
  (car (member-if (lambda (type-elt)
                    (or (eq type-elt type)
                        (and (listp type-elt)
                             (eq (car type-elt) type))))
                  (cdr link))))

(defun link-beginning (link)
  (second (link-type-accessor link 'passage)))

(defun link-end (link)
  (third (link-type-accessor link 'passage)))

(defun link-version (link)
  (second (link-type-accessor link 'version)))
\end{lisp}

\begin{notate}{Link creators} \label{link-creators}
It might be nice to have a function that is in charge
of link creation, so we can quickly identify the places
where links are being created.
\end{notate}

\begin{notate}{Link clusters} \label{link-clusters}
In previous versions of this work, I grouped links into ``segments''
by default.  I subsequently realized that adding this extra default
layer was ugly, and that it would limit the attractiveness of the
system.  Nevertheless, the initial intention was, I think, good.  The
idea with segments was to group certain links together semantically.
For example, if the region that a reference was attached to got split
in two after editing, one might want to maintain that one reference as
being \inp{about} two regions ``in the first place'' while continuing
to be about only the one target article ``in the second place''.
(On references, see Note \ref{references}.)  

In the case of references, we can get away with making making the
``link text'' correspond to all of the links besides the \emph{last}
one.  And in many other cases, similarly, we will be able to specify
semantics for \inp{about} data in the \inp{type} data.  However, in
some cases, we may need to put together link clusters that are
separate from the article entirely -- in other words, we need to be
able to attach scholia to \inp{about} data (as well as, e.g., to
\inp{text} data).

This flexibility assures us that there can be as many different kinds
of clusters as we can come up with.  By contrast, if we were using
segments, the actual semantics would either have to be more limited,
or would end up building extremely complex \inp{about} fields.

More work on the subject of link clusters will follow in due course.
Right now, I think they are just conceptually useful.  (Which is
another reason for getting rid of segments... thank goodness
\emph{they} never had a chance to become useful!)
\end{notate}

\begin{notate}{Uses for `type' field}
One use of types is to maintain subcollections; see Note
\ref{on-modified-type-labeler-etc}.  Articles of different types can
be rendered or otherwise interpreted differently.
\end{notate}

\begin{notate}{Hooks for new and modified scholia} \label{hooks-for-new-and-modified-scholia}
If the scholium is new, run `new-scholium-hook', otherwise, run
individual hooks depending on which pieces of the scholium were
modified (if any).  We use separate hooks in part because we need to
be careful about what sorts of changes we make.  If we adjusted the
value of some scholium every time the value of a scholium was
adjusted, we would expect to encounter infinite loops pretty quickly!

The standard settings for these hooks will be given in the section
\ref{subcollections}; these settings facilitate e.g. the automatic
creation of backlinks (Note \ref{backlinks}).
\end{notate}

\begin{lisp}
(defvar new-scholium-hook nil)
(defvar scholium-modified-text-hook nil)
(defvar scholium-modified-about-hook nil)
(defvar scholium-modified-type-hook nil)
(defvar scholium-modified-book-hook nil)
\end{lisp}

\begin{notate}{The `scholium' function} \label{the-scholium-function}
This function is foundational for the system.  Lots of other functions
will use it, but because it is destructive, users should typically not
access it directly.  Several interactive interfaces to this function
appear in Section \ref{creation}.  Note that the conditional structure
here shouldn't be a `cond'; we really mean to run several hooks if
several parts of the scholium have been modified.
\end{notate}

\begin{lisp}
(defun scholium (name text &optional about type book)
  (let ((old-version (get-article name)))
    (put-article name (list text about type book))
    (when (not (metadata-override))
      (if (not old-version)
          (run-hooks 'new-scholium-hook)
        (when (not (equal (scholium-text old-version) text))
          (run-hooks 'scholium-modified-text-hook))
        (when (not (equal (scholium-about old-version) about))
          (run-hooks 'scholium-modified-about-hook))
        (when (not (equal (scholium-type old-version) type))
          (run-hooks 'scholium-modified-type-hook))
        (when (not (equal (scholium-bookkeeping old-version) book))
          (run-hooks 'scholium-modified-book-hook))))))
\end{lisp}

\begin{notate}{On `metadata-override'} \label{metadata-override}
We don't want to run the usual hooks when the article being processed
is a metadata article or a ``fake'' article.  Fake articles appear
when we create fake references that they don't show up in the scholia
display and that aren't permanently attached to anything.  See Note
\ref{add-visible-back-temporal-link}.

(At least at one point I had some misgivings about this override, but
it seems to get the job done.  I don't see why we would want to have
weird automatic thing happen after creating a metadata article or a
``fake'' article, so it may be that the design here is perfectly
fine.)
\end{notate}

\begin{lisp}
(defun metadata-override ()
  (when (or (typedata-includes type 'meta)
            (typedata-includes type 'fake))
    t))
\end{lisp}

\subsubsection{Simple scholium derivatives}

\begin{notate}{On `article-names-from-about-data'}
The idea here is to build a list with just the names of the articles
that the \inp{about} data features.  Names are uniquified.
\end{notate}

\begin{lisp}
(defun article-names-from-about-data (about)
  (delete-dups (mapcar (lambda (elt) (car elt)) about)))
\end{lisp}

\begin{notate}{On `typedata-includes'} \label{typedata-includes}
Read \inp{typedata} in the format used in scholia,
and say whether it matches a \inp{specific-type}.
Notice that when \inp{typedata} and \inp{specific-type} are
both is `nil', we return a positive match.  
\end{notate}

\begin{lisp}
(defun typedata-includes (typedata specific-type)
  (cond ((eq typedata specific-type) t)
        ((and (listp typedata)
              (member specific-type typedata)) t)
        (t nil)))
\end{lisp}

\begin{notate}{On `typedata-includes-element-of-list'} 
Like
`typedata-includes' (Note \ref{typedata-includes}), but only
requires one type out of those listed to match in order to return
`t'.
\end{notate}

\begin{lisp}
(defun typedata-includes-element-of-list (typedata list-of-types)
  (let (ret)
    (while (and list-of-types
                (not ret))
      (when (typedata-includes typedata (car list-of-types))
        (setq ret t))
      (setq list-of-types (cdr list-of-types)))
    ret))
\end{lisp}

\begin{notate}{On `typedata-includes-passage'} \label{typedata-includes-passage}
A specific test to see if typedata includes type ``passage'' (noting
that the list element that indicates this type should come together
with some suffix that tells you which passage is being indicated).

As far as I can tell, this function will always apply to typedata
coming from links (but who knows).
\end{notate}

\begin{lisp}
(defun typedata-includes-passage (typedata)
  (let (ret)
    (when (listp typedata)
      (while (and typedata
                  (not ret))
        (when (eq (car (car typedata)) 'passage)
          (setq ret (car typedata)))))
    ret))
\end{lisp}

\subsubsection{Examples} \label{examples}

\begin{notate}{Examples of the `scholium' function in use}
As an example of how the `scholium' function might be applied, here is
a scholium about the article I'm writing now.
\begin{idea}
(scholium "Remark On Scholium Definition"
          "This is an Emacs Lisp function."
          '(((passage "sbdm4cbpp.tex" 49078 49738))))
\end{idea}
Actually, the article that I'm writing right now won't be in the
article list unless we explicitly put it there.  So the scholium
that we added here is actually attached to a \emph{fictitious}
article with the name ``sbdm4cbpp.tex''.  That's OK, but for
completeness, I'll add the current buffer as a scholium with the
appropriate name:
\begin{idea}
(scholium "sbdm4cbpp.tex"
          (current-buffer))
\end{idea}
Notice that we've supplied a buffer instead of a string for the
\inp{text} field.  We now add another scholium, attached to the
previous two, communicating to the user a fact that the system already
knows:
\begin{idea}
(scholium "This is a scholium about part of the SBDM document"
          "This scholium is attached to this region."
          '(((article "Remark On Scholium Definition"))
            ((passage "sbdm4cbpp.tex" 31091 31744))))
\end{idea}
By setting the \inp{type} data appropriately, we might make this last
example display when ``Remark On Scholium Definition'' is displayed,
but not when ``sbdm4cbpp.tex'' is displayed.  It might make sense to
mark up the words ``this region'' with a reference to
``sbdm4cbpp.tex''.  Tricky things like this will be coming in later
sections.
\end{notate}

\begin{notate}{Mixed scholia} \label{mixed-scholia}
Designing functions to build increasingly complicated \inp{about}
structures easily seems to become increasingly challenging.
\end{notate}

\begin{notate}{Types of text} \label{types-of-text}
There could be other kinds of ``text'' that one might want to be able
to use -- symbols, processes, web pages, etc.; basically, any sort of
information-containing thing could be something that one would wish to
add to the scholium system.  In addition, plain-text scholia can be
miniature database entries (in some label) if they are given a
specific \inp{type}.  For example, a function object might have a
\emph{descriptive note}, a \emph{doc string}, and a \emph{definition}.
Lists are another example of a special plain-text article
(see Note \ref{semantics-of-lists}).
\end{notate}

\subsubsection{Further notes on creating scholia}

\begin{notate}{The division of scholia into pieces}
The division is somewhat arbitrary.  Ownership could certainly be
maintained separately from edit history; however, it makes some sense
to set access controls when the article is first saved.  Things can
get complicated if different persons or groups have different
permissions; e.g. some may be able to change the access control list
and some may not.
\end{notate}

\begin{notate}{Completeness of the scholium function's arguments}
I think I've basically covered everything I'd be interested in with
these freeform types.  The idea is that you can put whatever you want
into these fields, and then write whatever sort of accessor functions
you like to use that information!  However, if someone wanted to add
some new fields, this would be easy; they would only have to change
the scholium function and the add-props function.
\end{notate}

\begin{notate}{How to assert that a relationship exists}
The \emph{relationship between} different articles presumably has to
be added with scholia that point to both articles and say something
special about them in the ``type'' field. On the other hand, it would
be useful to have that particular \inp{type} be read both ways, so
that scholia about both the high-level document and lower-level
document explain the relationship of the two documents to one another.
(I think backlink registries would help with this.)  We'll talk more
about how to actually assert relationships between articles in Section
\ref{editing}.
\end{notate}

\begin{notate}{Freedom to create derivative versions}
In general, this can be specified in the bookkeeping information -
i.e., the licensing terms (or royalty terms or whatever).  For just
about any article, it is important to be able to track derivatives!
The insistance on tracking makes our system not so different from
``regular'' version management.
\end{notate}

\begin{notate}{Types and modes}
Oftentimes, articles are not so different from buffers.  Type data is
a way of specifying things like filetype or editing mode.  (Maybe a
bit more general, because articles need not be very buffer-like.)
\end{notate}

\begin{notate}{Extended access}
It might be a good idea to let \inp{about} be filled by a
\emph{function} that will select a sequence of character positions?
Well, I suppose that even if the capability isn't explicitly coded
into the system, people can write their own code that does the job
(expands such functions before calling `scholium')?

Alternatively, if we had sub-libraries in this library, such
a function might be more complicated.  It would be worth thinking
about examples of how this sort of thing might be used, exactly.

Of course, if the library contains heterogeneous materials, like the
case that is being discussed in the ``superimposed information''
articles; there, selecting a region may mean different things for
different articles.  It seems like the case of Word docs and PDFs is
simpler than the case of whole libraries (e.g. filesets?) but I'd need
to think more about the issues here to come up with good examples.  In
which case would one rather add a library than all of the documents?
(Probably there are some cases.)  Well, I guess the GCIDE example that
I talked about could be relevant here, although the idea of this GCIDE
thing isn't quite the same as the usual web-like ``scholium thing''
we're considering elsewhere in this work.  Also, one might think of
Google or whatever as being a ``library'' that you have to interact
with in a special way to get at the documents -- the reasons for not
adding them are that they aren't free (for one thing) and there are
just too many of them (for another!).

Of course, scholia for really ``weirdly'' formatted documents --
including very rich things, like pictures or videos -- may have very
different sorts of semantics.  Its nice of the people who are working
on this ``superimposed information'' stuff to invest time and energy
into this sort of thing, but my own feeling is that its important to
get some good, working, deployments for text-based document
collections going asap!
\end{notate}

\begin{notate}{Type creation} \label{type-creation}
The AI issue relates to the issue of type creation.  We should be
talking about the relationship of semantic net ideas (an up-to-date
reference on semantic nets would be helpful; maybe SNePs).
\end{notate}

\begin{notate}{Formulaic scholia} \label{formulaic}
One obvious sort of type is the \emph{form}, i.e., an object with
certain fields to be filled in.  Compare
HyperCard\footnote{{\tt http://en.wikipedia.org/wiki/Hypercard}}; more
interesting ideas can be gotten there too, I think.
\end{notate}

\begin{notate}{Transformative scholia} \label{transformative}
It is interesting to think of transformative scholia; attachments that
change the way a given source looks.  This is a topic that has
recently been getting some play in the web browser community.  Simply
being able to annotate web pages would be swell, because then you
could search through them as if they were part of your own local web,
which, in fact, they probably would be.
\end{notate}

\begin{notate}{Functional scholia} \label{functional}
Functional scholia are similar to (and generalize) the transformative
scholia of Note \ref{transformative}.  Reimplementing LISP inside the
scholium system would be an example of this sort of thing; functions
and variables as nodes in a web, programs as clusters of nodes that
have been linked together in certain ways.  One way this might be done
would be to use the ordered \inp{about} list as an argument list for a
function and put the function itself on either the \inp{type} field
or, more likely, the \inp{text} field (a specification of the expected
arguments would go on the \inp{type} field).  Perhaps something like
this could be used to build documentation for LISP functions
automatically; needless to say, anything we can do scholia-wise can be
done LISP-wise (reasoning by set-inclusion, if nothing else), however
there may be some advantages to thinking about code scholiumifically.
(Functions and their subroutines can easily be regarded as forming a
network, and so, code can easily be thought of as a scholium-based
document under this arrangement.)
\end{notate}

\begin{notate}{Interactive and actionable scholia} \label{interactive-and-actionable-scholia}
One of the big goals for the system is to be able to write MUD-like
programs and text-based games; to do this we need, \emph{interactive}
scholia.  More generally, scholia that do something e.g. when browsed
or when encountered by a given process; we call elements of this
broader category \emph{actionable} scholia.
\end{notate}

\begin{notate}{Simulating features from single-buffer editing}
In Note \ref{functional}, we mentioned a certain feature that should be
replicated in the scholium system to make it more useful
(`get-function'); however, some systematic study of the features that
are useful from single-buffer editing should be made, and those
features re-implemented for the scholium system.
\end{notate}

\begin{notate}{Handling functions}
A simple feature request somewhat related Note \ref{functional} is
that we should be able to jump to the scholium associated with a
certain function's name; the analogue of my `get-function' command.
The text that documents a certain function should also show up as a
docstring when one runs `describe-function' in Emacs.
\end{notate}

\subsection{Metadata} \label{metadata}

\subsubsection{Metadata articles} \label{metadata-articles}

\begin{notate}{Metadata article overview} \label{metadata-article-overview}
Metadata articles constitute a merger between ``local'' and
``holographic'' data storage (Note \ref{holographic-data-storage}).
Metadata articles are essentially local, but they are used to maintain
information about data drawn from all over (in particular, backlinks
are recorded on metadata articles).  A metadata article is always
attached to some other article, and is used to record \emph{common}
information about this article.  They are updated automatically when
certain editing events take place.  When an article is rendered,
certain fields in the metadata article can take precedence over
corresponding information in the article itself (see Note
\ref{updating-link-extent}).
\end{notate}

\begin{notate}{Metadata article design} \label{metadata-article-design}
We exploit the facts that articles (a) have unique names, (b) any LISP
object can be the name of an article, to produce the rule that any
\emph{non}-metadata article with \inp{name} \emph{name} to which we
need to associate a metadata article will have a metadata article
named ``(meta \emph{name})''.  The \inp{text} of this article is the
metadata associated with the base article; its format is that of a
list of lists, with each sub-list headed by a tag describing the
contents of that sub-list (e.g. \emph{backlinks}).  Metadata articles
are not considered to be \inp{about} anything; we get the picture by
looking at the article's \inp{name}.  (They may however contain
updated \inp{about} information for the article they are ``meta'' to;
see Note \ref{updating-link-extent}.)  Also, for now at least, we give
them only very minimal \inp{bookkeeping} information.

A ``metadata'' label is maintained that allows us to easily keep track
of the collection of metadata articles (see Note
\ref{on-modified-type-labeler-etc}, \emph{et seq.}).

This design makes it easy to set metadata ``fields'' with `assoc' and
`setcdr'; see Note \ref{put-metadata-field}.

(It would be good to say a bit more about the different fields that
are stored in metadata articles here.)
\end{notate}

\begin{notate}{Maybe phasing out use of pseudo-namespace}
So far we've been able to ``afford'' the use of a pseudo-namespace for
metadata articles.  However, we may need to use a more legitimate
approach at some point.  We'll look at this issue again if it ever
seems important.
\end{notate}

\begin{notate}{On `metadata-article-name-p'}
So, like I said, we're losing some generality in the
set of available names by doing things the way we've been doing it.
\end{notate}

\begin{lisp}
(defun metadata-article-name-p (name)
  (when (and (listp name)
             (eq (car name) 'meta))
    t))

(defun base-name-from-metadata-name (name)
  (second name))
\end{lisp}

\begin{notate}{On `metadata-article'}
This function retrieves the metadata article associated with the
article named \inp{name}.
\end{notate}

\begin{lisp}
(defun metadata-article (name)
  (get-article (list 'meta name)))
\end{lisp}

\begin{notate}{On `put-metadata-field'} \label{put-metadata-field}
This can be can be used to fill a metadata \inp{field} with the given
\inp{value}, in the metadata article pertaining to the article named
\inp{name}.  If the field has not yet been set, it is created anew
with the appropriate \inp{value}.
\end{notate}

\begin{lisp}
(defun put-metadata-field (field value name)
  (let* ((metadata (metadata-article name))
         (old-value (assoc field (scholium-text metadata)))
         new-text)
    (if old-value
        (setq new-text (progn (setcdr old-value value)
                              (scholium-text metadata)))
      (setq new-text `((,field . ,value))))
    (scholium (list 'meta name)
              new-text
              nil
              'meta
              'system)))
\end{lisp}

\begin{notate}{On `get-metadata-field'}
This can be used to retrieve any particular metadata \inp{field}
pertaining to the article named \inp{name}.
\end{notate}

\begin{lisp}
(defun get-metadata-field (field name)
  (cdr (assoc field (scholium-text (metadata-article name)))))
\end{lisp}

\begin{notate}{Similarity between metadata and text properties}
It makes some sense that metadata about articles and metadata
about characters would be treated somewhat similarly.  Anyway,
the similar idioms seem to arise.
\end{notate}

\begin{notate}{Updating link extent} \label{updating-link-extent}
If an article $A$ is a scholium attached to some other article $B$,
and $B$'s \inp{text} is edited, then $A$'s \inp{about} data, if taken
literally, may be quite inaccurate.  The solution to this problem is
to examine the text properties indicating which regions of $B$ $A$ is
\emph{supposed to be} \inp{about}, and store the updated information
where it will be useful: $A$'s metadata article ``(meta $A$)''.  The
information stored in (meta $A$) ``masks'' individual links; see Note
\ref{masks} for more information.
\end{notate}

\begin{notate}{Updated `about' fields in the distributed case} \label{updating-link-extent--distributed-case}
In the distributed case, it would probably be best to use the metadata
pertinent to the current checked-out instance.  To facilitate this,
contributed \inp{about} data should probably be stored together with
the id of its source.  We're not quite there yet, but hopefully soon
will be!  (See also Note \ref{conflicts-in-a-distributed-environment}
\end{notate}

\subsubsection{Backlinks (who links here?)} \label{backlinks-section}

\begin{notate}{Theory of backlinks} \label{backlinks}
One particularly interesting sort of object is the ``backlink''.
These complement the generalized links discussed in Note
\ref{generalized-links}, and they play a fundamental role in the
system.  Backlinks make some operations faster (compare Note
\ref{name-or-access-buffered-articles}).

Unlike regular links, which live in articles, an article's backlinks
live in its ``public'' metadata article.  See
\ref{overview-of-linking-process} for further information on the
lifecycle of a backlink.

See Note \ref{backlink-tricks} for a description of tricks that can be
done with backlinks.  Its worth mentioning that backlinks not seen in
the typical display, but they could be used in special displays (see
Note \ref{ancestors}).  Backlinks are going to be important for AI
applications; they provide the information a given node needs to
``know'' its context within the broader semantic network.

Note that Howm is an already existing Emacs package for both forward
(``goto'') and backward (``come-from'') linking\footnote{{\tt
    http://howm.sourceforge.jp/}}.  The behavior of Howm's come-from
links is somewhat different from the backlinking features implemented
in the scholium system, but it shouldn't be too hard to add Howm-like
features (see Section \ref{otf} for some words on how we might
approach this).

(This note should probably just state the definition of backlinks;
anyway, as it is it seems to vague and too long.)
\end{notate}

\begin{notate}{Implementation of backlinks} \label{implementation-of-backlinks}
Whereas links are typed (Note \ref{links}), backlinks are not.
Insteady, backlinks indicate an article and a certain link in that
article.  The backlink data is useful because it allows us to easily
figure out which articles link to a given article (without doing any
mapping).  The basic use of backlink data is to display an article
together with its scholia.

We had considered using typed backlinks (mirroring the type of the
link that each backlink corresponds to), but this seemed excessive,
since the corresponding information can easily be found by examining
the link corresponding to a given backlink.
\end{notate}

\begin{notate}{Overview of linking process} \label{overview-of-linking-process}
When \inp{about} data is created or modified, `put-backlinks' is
called to update the backlinks in the metadata articles associated
with the \emph{linked-to} articles.  A backlink points at the specific
link that it corresponds to, by way of the \emph{linking} article and
the unique \emph{link number} that identifies the corresponding link.
\end{notate}

\begin{notate}{On `delete-backlink'} \label{delete-backlink}
This function is called by `put-backlinks' to prune away an old
backlink.

Also notice that in this function, we know that some
metadata and backlinks already exist, so in particular the
`metafield' exists.  Hence we don't need the complicated
conditions found in the `add-backlink' function.

(I don't suppose we actually need to delete the ``backlink'' field
in the case in which we delete the last backlink.)  
\end{notate}

\begin{lisp}
(defun delete-backlink (linked-to-article link-number)
  (let* ((metaarticle (metadata-article linked-to-article))
         (metatext (scholium-text metaarticle))
         (metafield (assoc 'backlinks metatext))
         (backlinks (remove (list name link-number)
                            (cdr metafield))))
    (setcdr metafield backlinks)
    (scholium (list 'meta linked-to-article)
              metatext
              nil
              'meta
              'system)))
\end{lisp}

\begin{notate}{On `add-backlink'} \label{add-backlink}
This function is called by `put-backlinks' to add a new backlink.

Note that the relevant metadata article may not exist prior to the
time this function runs and even if it exists, it may not contain any
backlink data.  So we have to run the relevant tests to deal with
these cases.  This accounts for differences between the implementation
of `add-backlink' and `delete-backlink' (Note \ref{delete-backlink}).
\end{notate}

\begin{lisp}
(defun add-backlink (linked-to-article link-number)
  (let* ((metaarticle (metadata-article linked-to-article))
         (metatext (scholium-text metaarticle))
         (metafield (assoc 'backlinks metatext))
         (backlinks (add-to-or-start-list
                     (cdr metafield)
                     (list name link-number))))
    (cond
     (metafield
      (setcdr metafield backlinks))
     (metatext
      (setcdr metatext
              `(((backlinks . ,backlinks)))))
     (t
      (setq metatext
            `((backlinks . ,backlinks)))))
    (scholium (list 'meta linked-to-article)
              metatext
              nil
              'meta
              'system)))
\end{lisp}

\begin{notate}{On `put-backlinks'} \label{put-backlinks} 
The basic use for this function is to run it within the scope of the
`scholium' function, to build and store new (or updated) backlink
data.  Backlinks are added to each of the metadata articles
corresponding to articles that this article is \inp{about}.

The algorithm used here (zap \emph{all} existing backlinks, then place
\emph{all} new backlinks) seems likely to be woefully inefficient!  If
so, this can be improved later.

Notice that we don't take into account \inp{name} changes with this
function (there is no relevant ``old-name'' variable, at least, not at
present).
\end{notate}

\begin{lisp}
(defun put-backlinks ()
  (let ((old-about (scholium-about old-version))
        (link-number 0))
    (dolist (link old-about)
      (setq link-number (1+ link-number))
      (delete-backlink (car link) link-number)))
  (let ((link-number 0))
    (dolist (link about)
      (setq link-number (1+ link-number))
      (add-backlink (car link) link-number))))

(add-hook 'new-scholium-hook 'put-backlinks)
(add-hook 'scholium-modified-about-hook 'put-backlinks)
\end{lisp}

\begin{notate}{Generality of `put-backlinks'} \label{generality-of-put-backlinks}
The function used to be written in a general way so that it can store
backlinks in different fields.  For example, I had set things up to
add to a `derives-to' metadata field when the function
`include-article' ran (see Note \ref{include-article}).  However, it
wasn't clear to me that this approach was really useful or necessary.

Nevertheless, storing new or updated mask data has to be done at some
point; and it may be that we'll want a generalized version of
`put-backlinks' after all (perhaps with a better name, speaking to its
new general purpose).
\end{notate}

\begin{notate}{Turning backlinking off on demand sometimes?}
It might be nice to turn backlinking off in certain situations.  A
rather silly example is the situation in which you don't want the
owner of a certain object to know that you are talking about it.
There may be some more relevant situations.  (Compare the use of fake
links, as documented in Note \ref{metadata-override}).
\end{notate}

\subsection{Consistency conditions} \label{consistency}

\begin{notate}{Why we maintain consistency} \label{consistency-why}
This discussion might go into Section \ref{philosophy} or Section
\ref{mathematics}, but given that it is fairly foundational for the
system, it is worthwhile to at least include an overview early on.

When something changes in the system -- whether it is the addition of
an article, or the deletion of an article, or edits to an article --
very often the effects are not purely local; or at least, they
shouldn't be.  Someone may want to know when a certain kind of article
is created.  Articles that referred to an article that gets deleted
may want to refer to something else.  If an article changes,
references to specific regions of that article may have to be
re-directed or, in some cases, eliminated.

Many of these operations can be taken care of automatically by a
sufficiently powerful hypertext system.  Indeed, particularly in the
case of multi-user editing, the consistency conditions (considered as
a very general category) essentially constitute the \emph{social
  contract} associated with the work.  
\end{notate}

\begin{notate}{When we maintain consistency} \label{consistency-when}
As we mentioned in Note \ref{consistency-why}, in the broadest sense,
we're always maintaining some sort of consistency.  But the three
major categories, \emph{adding}, \emph{deleting}, and \emph{editing},
give a nice first-run division of the cases.  (There may be some other
situations that will come up, and will not fit immediately into any of
these three groups, e.g. maintaining some state that isn't officially
part of an article per se, but at least ideally, we can decompose the
space this way.)

Sometimes users won't want certain ``weird'' things to happen to their
articles automatically.  For example, maybe someone really means to
point at the region between character position 5 and character
position 31 in some particular article, and not to the string that
currently appears in this region at the time when the link is created.
In this case, the person creating the link should say so -- and the
system should be set up to adhere to the reasonable wishes of its
users.  In particular, it should be entirely possible for two users to
have vastly different views of the document's contents!

(Notice that in the distributed case, it we may have to adopt somewhat
different notions of consistency; see Section \ref{distributed}.)
\end{notate}

\begin{notate}{Consistency conditions for adding} \label{consistency-adding}
This is a fairly simple case, relatively speaking.
It includes such things as automatic labeling
(cf. Section \ref{labels}) as well as the automatic addition
of backlinks to appropriate articles (Note \ref{put-backlinks}).
\end{notate}

\begin{notate}{Consistency conditions for deleting} \label{consistency-deleting}
The most important feature associated with deletion is to delete or
redirect backlinks.  See Section \ref{deletion} for details.
\end{notate}

\begin{notate}{Consistency conditions for editing} \label{consistency-editing}
This is by far the most complicated of the three cases.  When an
article is edited, links that point to its various components may have
to be updated.  In particular, when an article's \inp{about} data is
edited, anything that pointed at this data must change -- this
includes everything that uses link-id's (Note \ref{link-ids}), i.e.,
backlinks in other articles, masks in the linking article, and markup.

Changes to an article's \inp{text} can have repercussions of similar
complexity, especially when derivative works are taken into account;
see Section \ref{derivative} for details.
\end{notate}

\subsection{Subcollections} \label{subcollections}

\subsubsection{Labels} \label{labels}

\begin{notate}{Cost of maintaining labels} \label{cost-of-maintaining-labels}
To maintain a label, we have to do predicate testing as we go along.
This is a way of amortizing the cost of lookup.  (An example would be
nice, I'm sure.)  The thought is that some overall savings can be
obtained this way -- but in general they don't come automatically; we
have to be intelligent about limiting the tests we run (at least, I
don't think we get savings for free by doing things this way).  For
example, we don't add backlinks to metadata articles -- but there
should be other intelligent things to do too.  Another idea in a
similar theme is to have several functions for creating different
kinds of articles; these functions can instructions for applying
certain labels hard-coded into them.
\end{notate}

\begin{notate}{Labels as made up of trivial clusions}
Given that the design of labels stores a list of link-like elements in
the \inp{text} field of a article, and the fact that we would like to
have backlinks from the labeled articles to the label itself (Note
\ref{autobacklink-to-label}), it would appear to me that labels are
somewhat like trivial clusions (Note \ref{clusions}).  Assuming that
we are going to go ahead with the backlinking from labeled articles
(which seems to be useful when we think about exploring or exporting a
hierarchy), then we may want to treat labels in the same way we treat
clusions, or alternatively, switch to putting their contents onto the
\inp{about} field instead.
\end{notate}

\begin{notate}{Comparison with labels in other systems} \label{comparison-with-other-label-systems}
Various folksonomy-oriented environments allow people to use labels.
The usage in these systems is certainly similar to usage in Arxana,
however, I'm not completely sure that the ideas are the same.  This
would bear further investigation.
\end{notate}

\begin{notate}{Label terminology}
Following on Note \ref{comparison-with-other-label-systems}, it seems
possible that the term ``label'' would be somewhat confusing if the
label was thought of as a set, and not a list.  So to be clear, what
we really are doing is keeping track of various things on separate
lists.  (This is relevant e.g. for `label-article-insert-before', Note
\ref{label-article-insert-before}).
\end{notate}

\begin{notate}{On `modified-type-labels'} \label{on-modified-type-labeler-etc}
An alist of types to look for paired with the corresponding label to
apply.  Each new or modified article is tested to see if it matches
any of the given types; if so, then the corresponding label(s) will be
applied.  (Note that the ``label'' label is not treated specially.)
\end{notate}

\begin{lisp}
(defvar modified-type-labels '((nil . plain)
                               (label . label)
                               (meta . metadata)
                               (list . list)
                               (reference . reference)
                               (section . section)))
\end{lisp}

\begin{notate}{On `label-article'} \label{label-article}
This function is responsible for the mechanics of editing labels, and
creates them as needed.
\end{notate}

\begin{lisp}
(defun label-article (name article-label)
  (scholium article-label
            (add-to-or-start-list (scholium-text
                                   (get-article article-label))
                                  name)
            nil
            'label))
\end{lisp}

\begin{notate}{On `label-article-insert-before'} \label{label-article-insert-before}
Like `label-article' (Note \ref{label-article}), but takes an
additional argument \inp{before-this} to say which entry the new name
should be inserted just before.
\end{notate}

\begin{lisp}
(defun label-article-insert-before (name before-this article-label)
  (let* ((contents (scholium-text (get-article article-label)))
         (before-this-headed (member before-this contents)))
    (when before-this-headed
      (let ((len (- (length contents)
                    (length before-this-headed)
                    1)))
        (if (> len -1)
            (setcdr (nthcdr len contents)
                    (cons name before-this-headed))
          (setq contents (cons name before-this-headed))))
      (scholium article-label
                contents
                nil
                'label))))
\end{lisp}

\begin{notate}{On `put-type-labels'}
This function runs within the scope of `scholium'.  It might make more
sense to be able to add lots of different labels for a new article;
the `cond' function doesn't work that way however.
\end{notate}

\begin{lisp}
(defun put-type-labels ()
  (dolist (type-label modified-type-labels)
    (when (typedata-includes type (car type-label))
      (label-article name (cdr type-label)))))

(add-hook 'new-scholium-hook 'put-type-labels)
(add-hook 'scholium-modified-type-hook 'put-type-labels)
\end{lisp}

\begin{notate}{Generalizations of the autolabeler}
At the basic level, what we want is a way to maintain a list of
predicates, such that any scholium that satisfies a given predicate
gets the corresponding label.  We could maintain several such lists,
but it is (theoretically) important to be general, and the design
should be evaluated to determine how easy it is to add new predicates
and labels.
\end{notate}

\begin{notate}{Could take subcollection as optional argument}
In general, the functions that operate on all articles should probably
take a subcollection as an optional argument.  This might be a
namespace, or a label, or a list that has been assembled by some novel
means.  Some of our functions already have something like this
working, but it hasn't really been standardized in any way.
\end{notate}

\subsubsection{Namespaces} \label{namespaces}

\begin{notate}{Namespace implementation}
Namespaces have to be implemented in a nicely self-similar way.
Whereas an article can bear any number of labels, it can be only in
one namespace at a time.  (Since namespaces can be nested, we might
have $A\sqsupset B \ni a$, so that the article $a$ isn't a member of
$A$, but \emph{limiting} a recursive search to $A$ would still help
one to find $a$.)  

Actually, namespaces are a lot like the filesystem, so I can imagine
that we might provide the ability to symlink elements from one
namespace into another (similar to textual \emph{identification}, see
Section \ref{identification}).  So maybe best to take what I said
about only being in one namespace at a time with a grain of salt.
What is probably the case is that the object \emph{will} only be in
one namespace at a time, but that it can have ``virtual copies'' in
other namespaces.

If we go this route then we have two things to implement, roughly
self-similar namespaces themselves, and a method for doing
identification between them.  I hope we can follow (in one direction
or another) the implementation of \emph{labels}; we'll check that out
now.

The first step will probably be to make a namespace article-type, with
a specially formatted text field (a list or table), which is to be
displayed in a special way.  We'll most likely want a variant on
`read-article-name' that recurs if the name that is read in names a
namespace.  

Display and deletion will both have to have some work done on them to
make them namespace-compatible.
\end{notate}

\begin{notate}{Why namespaces} \label{why-namespaces}
Frequently one will be working with a data structure which has some
symmetries that you don't want to collapse.  For example, you might
have a \emph{Note} and a \emph{Section} with the same name in a given
article.  Or you might have several sections with the same name in
several different articles.  Or you might be working with a database
like structure in which every item has a \emph{foo}-type attachment,
and you want these to be represented as different scholia, not
\emph{one} scholium with type \emph{foo}.  Furthermore even in cases
where you aren't expecting overlap, it might be handy to keep
different collections strongly separated from each other.  (E.g. if
you have a bunch of articles coming from one resource and a bunch of
very different articles coming from another resource.)  In this case,
it is probably often just as good to use labels and keep the different
collections only weakly separated -- indeed, when there isn't overlap
in names, a sufficiently well-developed system for working with labels
will probably give you everything that you would have wanted from a
system with namespaces.  Be that as it may, overlap is prevalent
enough to make namespaces useful.  (Another approach to overlap is
\emph{disambiguation}, which is what Wikipedia uses; but having to use
different names all the time when disambiguating could get very
tedious.)
\end{notate}

\begin{notate}{Using namespaces for importing} 
In Note \ref{why-namespaces}, we mentioned that namespaces could be
useful when working with several articles with similar structure.
Underscoring that point, it seems we may typically want to use
namespaces when importing any secondary documents into the scholium
system (e.g. the HDM Manifesto or Noosphere's documentation).  They
may also be useful for articles that are stored remotely (e.g. we
might maintain one or more ``web'' namespaces).
3z  
z  
\end{notate}

\begin{notate}{Maintain the `label' article as a namespace?}
As one example, it might be advantageous to maintain the `label'
article as a namespace!
\end{notate}

\begin{notate}{On `generalized-put-article'} \label{generalized-put-article}
Like `put-article' (Note \ref{put-article}), but you specify
a \inp{path} as well as a \inp{name}.  The \inp{path}
should specify the path to the namespace wherein 
the \inp{name}, \inp{value} pair is to be stored.
\end{notate}

\begin{lisp}
(defun generalized-put-article (path name value)
  (puthash name value (generalized-get-article path)))
\end{lisp}

\begin{notate}{On `generalized-get-article'} \label{generalized-get-article}
Like `get-article' but takes a \inp{path}.  The path specifies a list
of nested namespaces, and possibly, a final non-namespace item to be
found in the last namespace.  Optional argument \inp{current} keeps
track of the ``current'' namespace as the path is digested.  If only
one item appears in the path and \inp{namespace} isn't given, the path
points to an item in the main article table, and we return the
corresponding item.

Be advised that there is a little bit of potentially destructive
ambiguity here.  But to make up for it, notice the doubly-clever use
of `and' and `cond' (to deal with the case that we're generalizing).
\end{notate}

\begin{lisp}
(defun generalized-get-article (path &optional current)
  (let ((first-step (get-article path)))
    (cond 
     ((and (not current) first-step)
      first-step)
     ((cdr path)
      (if (typedata-includes (get-article (car path)) 'namespace)
          (generalized-get-article (cdr path) (car path))
        (error "Path element not a namespace")))
     ((and (car path) current)
      (gethash current (car path)))
     ((car path) (get-article (car path))))))
\end{lisp}

\begin{notate}{Self-symmetry}
If we're going to have a set-up where things look roughly the same on
every level, for one thing, we may going to want to have a ``current
path'' variable, so we can move from level to level and populate that
level properly.  

We may want to have a different class of labels at every level in the
hierarchy.  Whether we could also have labels that reach across levels
is another question.  And as for metadata articles, should they be
stored in the top level or should they be stored in the same namespace
as the article that they apply to?  Storing them in the same namespace
\emph{sounds} good (it seems ``more distributed''), however remember
that it may not always be possible to access a given namespace to
store metadata.  Perhaps the best option is to mirror whatever
distributed resources one draws upon in the local implementation; then
storing metadata in the namespace local to the article being treated
would make some sense.  Or we could have one single namespace for
metadata articles (I've considered this option before) which would
take data coming from all other namespaces.  Seems like an interesting
approach, too.  These matters will require more thought.

Notice that with namespaces we're pretty likely to end up repeating
some of the work people have done with file systems.  On the other
hand, it is probably possible to use a simplified approach, with a
function to accomplish ``add article to namespace at the end of this
path''; like working with the file system but always from the root
directory.  Eventually we can add the convenience functions (really
not that complicated) to prepend a default path to the root.  (We seem
to already have the non-interactive elements of this arrangement with
`generalized-put-article' (Note \ref{generalized-put-article}) and
`generalized-get-article' (Note \ref{generalized-get-article}).)
\end{notate}

\subsection{Data access and conversion}

\begin{notate}{Introduction to data access and conversion}
The functions in this section give easy access to things we're often
interested in later on.  The functions here are simple, but it is
useful to have names for them anyway (Note
\ref{principle-of-transparency}).
\end{notate}

\begin{lisp}
(defun label-to-list (label)
  (mapcar (lambda (name)
            (format "%s" name))
          (scholium-text (get-article label))))

(defun label-to-propertized-list (label)
  (mapcar (lambda (name)
            (propertize (format "%s" name) 'name name))
          (scholium-text (get-article label))))
\end{lisp}

\begin{notate}{On `get-backlinks'} \label{get-backlinks}
This code produces the backlinks associated with \inp{name}.  See Note
\ref{implementation-of-backlinks} for details.  Notice that throughout
the code I have been applying this idiom
\begin{idea}
(lambda (backlink)
  (get-article (car backlink)))
\end{idea}
to the element of the returned list in order to come up with the names
of the backlinked articles.  However, I recently converted the
backlink format to be richer than it was before so that we knew
\emph{which link} was doing the linking, not just which article.  So
presumably we should be using this additional data at least some of
the time.
\end{notate}

\begin{lisp}
(defun get-backlinks (name)
  (get-metadata-field 'backlinks name))

(defun get-links (name)
  (cdr (assoc 'links (scholium-text (metadata-article name)))))
\end{lisp}

\begin{notate}{On `read-article-name'}
We frequently have to read the name of an article from a list.  We may
be able to do this in several different more intelligent ways than the
one we have here!  For one thing, we could use
`turn-article-table-into-names' (which see).  Another thing to do
would be to allow some other source of input, for example, some
particular namespace could be specified (this might be best handled
with another function, `read-article-name-from-namespace', say).  We
could offer some recursion if a namespace is selected.
\end{notate}

\begin{lisp}
(defun read-article-name ()
  (let* ((completion-ignore-case t)
         (strings (turn-article-table-into-list))
         (string-found (completing-read
                        "Article: "
                        strings))
         (place (- (length strings)
                   (length (member string-found strings)))))
    (nth place (turn-article-table-into-names))))
\end{lisp}

\begin{notate}{On `read-article-path'}
Like `read-article-name' but reads a path to an article through
namespaces (cf. `generalized-get-article', Note 
\ref{generalized-get-article}).
\end{notate}

\begin{lisp}
(defun read-article-path (&optional namespace path)
  (let* ((completion-ignore-case t)
         (strings (if namespace
                      (turn-namespace-into-list namespace)
                    (turn-article-table-into-list)))
         (string-found (completing-read
                        "Article: "
                        strings))
         (place (- (length strings)
                   (length (member string-found strings))))
         (ret (nth place (if namespace
                             (turn-namespace-into-names namespace)
                           (turn-article-table-into-names)))))
    (setq path (append path (list ret)))
    (if (and (typedata-includes ret 'namespace)
             (y-or-n-p "Read further? "))
        (read-article-path namespace path))
    path))
\end{lisp}

\begin{notate}{Reading from non-default namespaces} \label{read-from-non-default-namespace}
Perhaps there should be a ``read from this namespace'' variable that
can be swapped for the default in `read-article-name'.  This sort of
overload system should probably be used whenever we are reading from a
list of articles (just for example, see Note
\ref{make-reference-in-current-article}).

An alternative perspective on this matter is that it depends on the
behavior of `get-article'.  That function is responsible for finding
the article with the given name, regardless of where it resides.  So,
`get-article' needs to know where to look.  Perhaps `name' will
contain some instructions, or, more likely, perhaps this function
should take an optional argument that would help it to limit the
search.
\end{notate}

\begin{notate}{On `link-about-article-p'} \label{link-about-article-p}
This provides a quick way to tell whether one of the elements of an
\inp{about} list is actually about the article named
\inp{article-name}.
\end{notate}

\begin{lisp}
(defun link-about-article-p (link article-name)
  (equal (linked-to-article link) article-name))
\end{lisp}

\subsection{Further notes on scholia-based documents}

\begin{notate}{Following the design of metadata articles to make namespaces?} \label{metadata-design-for-namespaces}
Note that we could in theory follow a similar pattern with the meta
prefix to make additional ``namespaces'', i.e., the article ``(foo
\emph{bar})'' would be the article named ``bar'' within the namespace
``foo''.  We would maintain a label on these articles, as with the
``metadata'' label.  But it seems more desirable to create a namespace
\inp{type}, elements of which would have the property that they could
contain articles which aren't elements of the main article list.
\end{notate}

\begin{notate}{Design of real namespaces}
We currently only have one real ``namespace'' (see Note
\ref{metadata-design-for-namespaces} for a way to create a fake
namespace).  A real namespace would probably have a list or a hash
table in the \inp{text} field and a ``namespace'' token into the
\inp{type} field.
\end{notate}

\begin{notate}{Namespace narration}
Presumably a lot of verbose discussion of namespaces can be trimmed
away when we actually have an implementation.
\end{notate}

\begin{notate}{Finding namespaces by searching the namespace hierarchy}
We should sometimes be able to find namespaces, by searching through
the namespace hierarchy for their name.  (I think I may have recently
seen an Emacs package posted to gnu.emacs.sources for doing something
similar with directories or files.)
\end{notate}

\begin{notate}{Versioned backlinks?}
Note that backlinks are outside of the editorial control of the person
whose article is being linked to.  Nonetheless, it may be handy to
maintain versioned backlink repositories, possibly to correspond to
different versions of linked-to and the linked-from articles (this
could be very complicated).
\end{notate}

\begin{notate}{Holographic data} \label{holographic-data}
There are a number of different strategies for working with
``holographic data'', i.e., data that represents a document but that
contains different bits and pieces all over, which must be assembled
to construct any particular view into the document.  Some of these
strategies involve maintenance of datastructures and some involve
search.  (See Note \ref{holographic-data-storage} for some further
comments.)  For a pictoral illustration of something vaguely similar,
check out `animate-string'.
\end{notate}

\begin{notate}{Simulating a unified catalog using many namespaces} \label{namespaces-vs-catalogs}
The argument for namespaces says that we could simulate a full catalog
by having functions that trace through all of the namespaces and
gather all of the articles that have been found there.
\end{notate}

\begin{notate}{Are namespaces out-modded by labels?} \label{namespaces-outmodded}
I'm not even sure if we should be using namespaces at all, but one
category of cases that seems to motivate their use that in which you
have objects that contain sub-objects.  Namespaces seem like a natural
model, even if we decide never to use them explicitly.
\end{notate}

\begin{notate}{Remote subcollections}
We might want to have one subcolection for each source of articles in
a distributed system.  For a built-in web browser, we might also want
to maintain directions for re-loading everything that's been
\emph{downloaded} from the internet, probably as its own subcollection
(with instructions on where to find the articles on disc and
remotely).
\end{notate}

\begin{notate}{Holographic data storage} \label{holographic-data-storage}
This is the idea of a document in which data is stored ``all over'',
and our views into the document get different bits and pieces of this
data.  For example, in Table \ref{the-quick-brown-fox}, the string
``the quick brown fox'' is stored in a somewhat non-intuitive way,
namely, indexed by letter of the alphabet rather than character
position.  If we look at a neighborhood of the letter ``o'' (in the
natural metric coming from this index), we might see something like
\begin{center}
????q???k??ro?n??o?
\end{center}
which gives us some idea of what the document as a whole has to say.
(As would any neighborhood of any element, hence the appellation
``holographic''.)

Similarly, an article displayed together with its scholia gives a view
into the document it is a part of.  In general, its scholia may come
from ``all over''.  We use backlinks (Note \ref{backlinks}) to speed
the location of scholia necessary for presenting such views.

(Perhaps something from the theory of labels or something similar can
be used to speed things up too; e.g., I suppose that in addition to
generating this table, we could generate a list of the words in the
text too.  Something to think about I guess.)

\begin{table}
\begin{center}
\begin{tabular}{l|l}
{\bf character} & {\bf positions}  \\
\hline
 spc & 4, 10, 16\\
 b   & 11\\
 c   & 8\\
 e   & 3\\
 h  & 2\\
 i  & 7\\
 k  & 9\\
 n  & 15\\
 o  & 13, 18\\
 q  & 5\\
 r  & 12\\
 t  & 1\\
 u  & 6\\
 w  & 14\\
 x  & 19
\end{tabular}
\end{center}
\caption{\label{the-quick-brown-fox}Lexicographic storage for ``the quick brown fox''}
\end{table}
\end{notate}

\begin{notate}{World-writable components versus metadata articles} \label{metadata-article-permissions}
If we're going to maintain a collection of backlinks for each
linked-to article (which seems to be an exceptionally good idea), we
have to decide whether to maintain such a list as a part of the
article itself or as part of some other article (presumably an article
that contains nothing but metadata about the article that we're
working with).  This is an aesthetic consideration -- we're interested
in maintaining \emph{ownership} over articles, so for this reason, we
don't want them to contain world-editable components.  However, it may
not be so nice to have the article list filled up with system-owned
metadata articles, either.  Perhaps this is one case in which it would
be good to use a separate namespace.  (It is even conceivable that
users would want different behavior in different circumstances; e.g.
if all the articles are owned by one person, making portions of the
articles ``world-editable'' may not be such a problem.)
\end{notate}

\begin{notate}{Savings afforded by backlinks}
Here we assume that we use metadata articles of Note
\ref{metadata-article-permissions} and that we use a hash table to store the
document. If we maintain a list of backlinks for each article, the
added look-up cost of adding a link is $O(1)$ per linked-to article
(i.e. we need to find the appropriate place to store the backlink
metadatum), and the cost to display an article is $O(1)$ per
linked-from article (i.e. we use the list of backlinks to find each
scholium attached to the article being displayed).  On the other hand,
if we don't maintain backlinks, there is no look-up cost associated
with adding a link, but there is a cost of $O(\#\mathrm{Articles})$
associated with document display, since we have to test every article
to see if it is \inp{about} the article being displayed.
\end{notate}

\begin{notate}{Private information}
This is one potential use of namespaces (we could have a public and a
private namespace... though actually this sounds a bit more like
something that we would want to use \emph{labels} for).  Privacy is
generally considered to be an important right.
\end{notate}

\begin{notate}{Reading articles into a given namespace}
If a bunch of scholia have been saved to a file, it would be
convenient to have a way to read them all into one specific namespace.
\end{notate}

\begin{notate}{Intelligent search} \label{intelligent-search}
One topic we must concern ourselves with from time to time is the
\emph{scope of propagation}.  In a large system, we don't want to have
to look at everything after any given change.  Even if we maintain a
list of backlinks (see Note \ref{backlinks}), we might well have to
search through everything in order to find the articles that are
backlinked to!  (Which may not be such a bad thing if we use a hash
table.)  Another possible solution is to try to search intelligently,
by using namespaces to guide the search.  If each backlink tells you
roughly where the corresponding linking article can be found, then you
can recover that article much more efficiently.  I'm pretty sure this
relates to the system for IP addressses used on the web and also to
the ``humber'' system used in Xanadu.  (The Wikipedia page about the
IP system probably contains important background).  See also Note
\ref{metadata-article-overview} for related thoughts on using
namespaces to limit search.
\end{notate}

\begin{notate}{Possible relationship between labels and `type' data}
The \inp{type} field complements \emph{labels}; labels can be applied
by anyone, but type data can only be changed by the object's owner.
When you render an article, you look on its \inp{type} field first to
get the basic type data specified by the author.  Then you look at its
metadata article to find backlinks; if any of these backlinks are to
labels, you may find additional rendering information there.
\end{notate}

\begin{notate}{Movable scholia} \label{movable-scholia}
At various places in the lifecycle of a scholium, we'll want it to
move around, either concretely (from one base of attachment to
another) or ephemerally (appearing in some new place while continuing
to be attached in its original location).  We already have various
features associated with some of these processes, for example,
transclusion (Section \ref{transclusion}) is about viewing a given
article through some new window.  However, we'll want to do more,
e.g., one thing that should be coming soon is the ability to pass
scholia ``through'' a compilation (Section \ref{quick-compilations}).
We'll eventually want to be able to do the reverse as well.  In
general, support for the various \emph{concrete} operations
(e.g. passing a scholium through a reference and making it stick) are
lacking at present.
\end{notate}

\section{Adding articles to the digital library} \label{creation}

\begin{notate}{Introduction to adding articles in the digital library}
The code in this section provides various wrappers for the `scholium'
function, making it easy to create scholia.  Section \ref{bookkeeping}
details the process of adding appropriate \inp{bookkeeping}
information to new articles; articles created using the functions in
this section have standard \inp{bookkeeping} information added to them
automatically.  Section \ref{adding-text} concerns adding scholia
based on already-existing sources of text.  Section \ref{interactive}
presents functions that the user would call to create articles in
which the \inp{text} field is filled in by the user.

For \emph{editing} articles that have already been added to the
library, see Section \ref{editing}.
\end{notate}

\begin{notate}{Setting `type' field at creation time}
The user might want to set the \inp{type} field at creation time, not
just \inp{text} and \inp{about} (indeed, we could probably do a better
with \inp{about} in a few places, e.g. throughout Section
\ref{adding-text}).  But if it is not provided immediately, \inp{type}
can still be edited in later.
\end{notate}

\subsection{Supplying initial bookkeeping information} \label{bookkeeping}

\begin{notate}{On `sch-book'} \label{sch-book}
Interactive functions that call `scholium' should always provide
\inp{bookkeeping} information.

The following definition can be extended to add any sort of initial
bookkeeping information we might like to maintain.  The
\inp{bookkeeping} field will be revised when the article is edited
(see Section \ref{editing}).
\end{notate}

\begin{lisp}
(defun sch-book ()
  `((created ,user-login-name ,(current-time-string))))
\end{lisp}

\subsection{Adding text from an existing source} \label{adding-text}

\subsubsection{Adding text from buffers} \label{adding-buffers}

\begin{notate}{On `make-current-buffer-into-article'}
We store the object representing the buffer in the \inp{text} field.
There is no particular reason to set the \inp{type} (at least, one has
yet to appear).

An alternative approach would be to make an article's \inp{text}
reflect the current contents of the buffer (i.e., save the buffer as a
string) -- but we currently don't do this.  However, it would probably
be nice to have that option, or even just the option to make a certain
selection into an article.
\end{notate}

\begin{lisp}
(defun make-buffer-into-article (&optional buffer name)
  (let* ((buffer (or buffer
                     (get-buffer (read-buffer
                                  "Buffer: "
                                  (buffer-name
                                   (current-buffer))
                                  t))))
         (name (or name
                   (read-string "Name: "
                                nil
                                nil
                                (buffer-name buffer)))))
    (scholium name buffer nil nil (sch-book))))

(defun make-current-buffer-into-article (name)
  (interactive (list (read-string
                      (concat "Name (default "
                              (buffer-name (current-buffer)) "): ")
                      nil
                      nil
                      (buffer-name (current-buffer)))))
  (make-buffer-into-article (current-buffer) name))
\end{lisp}

\begin{notate}{Arbitrary buffer over current buffer?}
We could write `make-buffer-into-article' and
`make-current-buffer-into-article' in such a way that the former
calls the latter and uses `save-excursion'.  Both approaches
seem to have the same end result.
\end{notate}

\subsubsection{Adding text from files} \label{adding-files}

\begin{notate}{On `make-file-into-article'}
This function adds a file to the article list.  Again, the \inp{text}
field is not filled in with a string representing the text directly,
but rather, with the name of the file that holds the text.  A token is
added to the \inp{type} field explaining that this is a file.
\end{notate}

\begin{lisp}
(defun make-file-into-article (path name)
  (interactive
   (let* ((bufn (buffer-file-name))
          (pth (read-file-name
                (if bufn
                    (concat "File (default: " bufn "): ")
                  (concat "File: "))
                nil
                (if bufn
                    bufn
                  (default-directory))))
          (dnme (file-name-nondirectory pth))
          (nme (read-string (concat "Name (default: " dnme "): ")
                            nil
                            nil
                            dnme)))
     (list pth nme)))
  (scholium name path nil '(file) (sch-book)))
\end{lisp}

\subsection{Interactively supplying text} \label{interactive}

\begin{notate}{Global varibles describing new scholia}
The variables `new-scholium-name' and `new-scholium-about' are used to
build new articles interactively.  For now, \inp{type} is ignored by
these functions.  Also, we don't have a `new-scholium-text' field,
since we get the text another way.  It could be that an approach with
a `new-scholium-text' variable would allow us to unify the treatment
here with the one used in previous subsections.
\end{notate}

\begin{lisp}
(defvar new-scholium-name nil "Name of our new scholium.")
(defvar new-scholium-about nil "What the new scholium is about.")
\end{lisp}

\begin{notate}{On `new-scholium-mode'}
This mode is invoked by `make-scholium' in the buffer in which the new
scholium's \inp{text} is to be supplied.
\end{notate}

\begin{lisp}
(define-minor-mode new-scholium-mode
  "Mode for composing a new scholium.
\\{new-scholium-mode-map}"
  :init-value nil
  :keymap '(("\C-c\C-c" . escape-scholium-creation)))
\end{lisp}

\begin{notate}{On `escape-scholium-creation'}
Once the new scholium's \inp{text} has been supplied, this function
creates a scholium from that text and other data the user has
supplied.  It then restores the window configuration that was active
before `make-scholium' ran.  It also nullifies `new-scholium-name'
and `new-scholium-about', since we're done with these things.
\end{notate}

\begin{lisp}
(defun escape-scholium-creation ()
  (interactive)
  (scholium new-scholium-name
            (buffer-substring-no-properties (point-min) (point-max))
            new-scholium-about
            nil
            (sch-book))
  (kill-buffer (concat "Editing scholium: " new-scholium-name))
  (set-window-configuration sch-win-config)
  (setq new-scholium-name nil
        new-scholium-about nil))
\end{lisp}

\begin{notate}{On `make-scholium'} \label{make-scholium}
This function is called every time the user makes a scholium with new
text (i.e. text that is typed in on the fly).  Functions for making
scholia about articles, parts of articles, buffers, etc., are given in
this document and all use this function.
\end{notate}

\begin{lisp}
(defun make-scholium ()
  (setq sch-win-config (current-window-configuration))
  ;; we allow this to be set elsewhere
  (unless new-scholium-name
    (setq new-scholium-name (read-string "Scholium name: ")))
  (set-buffer (get-buffer-create "Scholia Display"))
  (other-window -1)
  (split-window-vertically)
  (other-window 1)
  (switch-to-buffer (get-buffer-create (concat "Editing scholium: "
                                               new-scholium-name)))
  (new-scholium-mode))
\end{lisp}

\begin{notate}{Redisplay after running `make-scholium'} \label{redisplay-after-running-make-scholium}
It would probably be fairly convenient to redisplay whatever object
was being displayed after `make-scholium' has run.  (It would be nice
to have the configuration of windows be 100\% consistent!)  In order
to make this be efficient, we would probably want to a system for
incremental markup, as described in Note \ref{incremental-markup}.  Presumably this just means
adding `redisplay-article' to an appropriate hook.
\end{notate}

\begin{notate}{Add a `make-scholium-hook'?}
We could add a hook here if we wanted to make it easy to do
interesting things after a scholium is added, like update the display.
(We might want to make a section on default settings -- and we might
want to make more hooks throughout the scholium system code.)
\end{notate}

\subsubsection{Scholia attached to the current article} \label{attached-main}

\begin{notate}{On `make-scholium-about-current-article'}
This function makes a scholium about the article as a whole.
\end{notate}

\begin{lisp}
(defun make-scholium-about-current-article ()
  (interactive)
  (when name-of-current-article
    (setq new-scholium-about `(((,name-of-current-article))))
    (make-scholium)))
\end{lisp}

\begin{notate}{On `make-scholium-about-part-of-current-article'} \label{make-scholium-about-part-of-current-article}
This function makes a scholium about one specific portion of the
article.

This function makes the (somewhat unrealistic seeming) assumption that
the current article and the current buffer are the same thing.  This
situation should be resolved.

However, if no article is current yet, then perhaps we should
offer to make a scholium about the current buffer?
\end{notate}

\begin{lisp}
(defun make-scholium-about-part-of-current-article (beg end)
  (interactive "r")
  (if name-of-current-article
      (progn
        (setq new-scholium-about
              `((,name-of-current-article
                  (passage 
                   ,beg
                   ,end))))
        (make-scholium)
        (deactivate-mark)
        (message (concat (format "%s--%s" beg end) " added.")))
    (message "Make some article current first.")))
\end{lisp}

\begin{notate}{The use of `deactivate-mark' in this code}
Since regions are being selected interactively, unless
we `deactivate-mark', we will be left with a highlighted
region after some of the scholium-creating function have
run.  This doesn't seem clean to me.
\end{notate}

\begin{notate}{On `make-scholium-about-current-line'}
Here is a little convenience wrapper for working with lists.
\end{notate}

\begin{lisp}
(defun make-scholium-about-current-line ()
  (interactive)
  (make-scholium-about-part-of-current-article (line-beginning-position)
                                               (line-end-position)))

(defun make-scholium-about-current-line-quickly ()
  (interactive)
  (setq new-scholium-name (buffer-substring-no-properties
                           (line-beginning-position)
                           (line-end-position)))
  (make-scholium-about-part-of-current-article (line-beginning-position)
                                               (line-end-position)))

(defun make-scholium-about-current-line-quickly-and-completely ()
  (interactive)
  (scholium (buffer-substring-no-properties
                           (line-beginning-position)
                           (line-end-position))
            (buffer-substring-no-properties
                           (line-beginning-position)
                           (line-end-position))
            `((,name-of-current-article 
               (passage
                ,(line-beginning-position)
                ,(line-end-position))))
            nil
            (sch-book)))

(defun make-scholium-about-current-sentence ()
  (interactive)
  (make-scholium-about-part-of-current-article (line-beginning-position)
                                               (line-end-position)))
\end{lisp}

\begin{notate}{Semantics of lists} \label{semantics-of-lists}
Lists may benefit some special semantics.  In particular, it might be
useful to only display the scholia that are attached to \emph{the
  current line} (or item more generally).  Also, an indicator to show
the existence of scholia attached to other lines (when such scholia do
in fact exist).
\end{notate}

\begin{notate}{Scoring}
Maybe the way to go with scoring would be to make a change to the
\inp{type} field to indicate that the thing has type ``score 10'' or
whatever.  Or perhaps we would want to attach a scholium?
\end{notate}

\begin{notate}{Locally-stored data versus indexed data} \label{locally-stored-vs-indexed}
This reminds me of the different models of data storage.  We could
keep everything local (a score field for each scholium) or store
things in an index (look-up each scholium to see if it has a score).
The magic of hash tables says that lookup is ``linear'' but is the
factor lower if the data is kept locally?  Maybe on average.

Of course, if you have to search a collection to find the extent of
some predicate, that will take \emph{some} time; if everything
matching a certain description has been stored on a certain list, then
you already have the predicate's extent without search.

There's really no reason not to store some things locally if we know
that search is going to always be local (e.g.  the score of a certain
article is probably fine to store locally) \emph{and} we have
permission to store that data locally.  See Note
\ref{holographic-data} for more on this general topic -- the issues
here have been pretty much resolved with in the design of metadata
articles (Note \ref{metadata-article-overview}).
\end{notate}

\begin{notate}{Local list of scholia}
One example of how local lists could come in handy (like we were
talking about in Note \ref{locally-stored-vs-indexed}) would be to
keep a local list of all the scholia that are attached to the current
article.  Then, if changes have to be propagated, we just look at the
things on this list, rather than having to search through all of the
articles again to figure out which scholia are attached to the current
article.  Of course, the local list could be kept updated as the set
of scholia attached to the current article changes.  This would (I
think) be an optimization, and isn't essential for building a working
system.
\end{notate}

\subsubsection{Creating a scholium that applies to several regions}

\begin{notate}{On `reading-regions-mode'}
This mode is invoked by
`make-scholium-about-several-parts-of-current-article', and
adds an editing mode in the buffer containing the current
article that enables the user to select regions that the
scholium will be \inp{about}.
\end{notate}

\begin{lisp}
(define-minor-mode reading-regions-mode
  "Mode for reading in regions.
\\{new-scholium-mode-map}"
  :init-value nil
  :keymap '(("\C-c\C-c" . add-region)
            ("\C-c\C-g" . escape-reading-regions-mode))
  (message "C-c C-c to add regions; C-c C-g to make scholium."))
\end{lisp}

\begin{notate}{On `add-region'}
This function adds regions to `new-scholium-about'.
\end{notate}

\begin{lisp}
(defun add-region (beg end)
  (interactive "r")
  (setq new-scholium-about
        (cons `(,name-of-current-article
                 (passage 
                  ,(region-beginning)
                  ,(region-end)))
              new-scholium-about))
  (deactivate-mark)
  (message (concat (format "%s--%s" beg end) " added.")))
\end{lisp}

\begin{notate}{On `escape-reading-regions-mode'}
When all of the regions desired have been selected, this function
calls `make-scholium' to finish things off.
\end{notate}

\begin{lisp}
(defun escape-reading-regions-mode ()
  (interactive)
  (reading-regions-mode -1)
  (make-scholium))
\end{lisp}

\begin{notate}{On `make-scholium-about-several-parts-of-current-article'} \label{on-make-scholium-about-several-parts-of-current-article}
This function makes a scholium that applies to several portions of the
article, using the mode and so on featured in this section.

(Note, it doesn't seem that this is displayed quite right; I'm
getting two copies of the scholium's text in the Scholia
Display buffer.)
\end{notate}

\begin{lisp}
(defun make-scholium-about-several-parts-of-current-article ()
  (interactive)
  (let ((article (get-article name-of-current-article)))
    (if (article-buffered article)
        (switch-to-buffer (get-buffer (scholium-text article)))
      (switch-to-buffer "Main Article Display"))
    (setq new-scholium-about nil)
    (reading-regions-mode 1)))
\end{lisp}

\subsubsection{Scholia about the current buffer} \label{about-buffer}

\begin{notate}{Making scholia about the current buffer}
We present facilities for making new scholia about the current buffer
or file that this buffer is visiting.  Corresponding functions for
displaying scholia that are attached to the current buffer appear in
section \ref{rendering}.  This function ensures that the buffer is
associated with on article on the article list; otherwise, we can't
make a scholium about it.
\end{notate}

\begin{notate}{On `call-if-user-adds-current-buffer-to-article-list'}
This is used by functions that require the current buffer to
be an article; typically they recall themselves after the
buffer has been added.  It is used by
`make-scholium-about-current-buffer' and
`display-scholia-about-current-buffer'.
\end{notate}

\begin{lisp}
(defun call-if-user-adds-current-buffer-to-article-list (fct)
  (when (y-or-n-p "Buffer not an article, add to list? ")
        (make-current-buffer-into-article
         (read-string (concat "Name (default: "
                              (buffer-name
                               (current-buffer)) "): ")
                      nil
                      nil
                      (buffer-name (current-buffer))))
        (funcall fct)))
\end{lisp}

\begin{notate}{On `make-scholium-about-current-buffer'}
This function makes a scholium about the current buffer,
requiring that it be an article.  (Maybe we should
just add the current buffer to the article list
transparently, rather than giving the prompt in
`call-if-user-adds-current-buffer-to-article-list'.)
\end{notate}

\begin{lisp}
(defun make-scholium-about-current-buffer ()
  (interactive)
  (let ((article (get-article (buffer-name (current-buffer)))))
    (if (not article)
        (call-if-user-adds-current-buffer-to-article-list
         'make-scholium-about-current-buffer)
      (setq new-scholium-about
            `(((,(buffer-name (current-buffer))))))
      (make-scholium))))
\end{lisp}

\begin{notate}{Scholium creation modalities} \label{scholium-creation-modalities}
We developed all sorts of creation facilities for generic articles,
shouldn't we have them all over again for buffers?  It seems only fair
that there would be some symmetry between the way articles and buffers
are handled.  Maybe there is some way to reuse the work.

On the other hand, maybe we don't need a lot of complex functions for
specially handling buffers, since once the buffer has been made into
an article, all of the functions for working with articles apply.
(Contrast Note \ref{bias-towards-buffers}!)
\end{notate}

\begin{notate}{Making a scholium about a file.}
Doing something like what `make-scholium-about-current-buffer' does
that makes a scholium about the current buffer \emph{as} a file
shouldn't be very different at all.  I'll want to think a little more
about the relevance of doing things this way: it may not really matter
if we run the exporting and importing routines just right.  (We could
also write a function to make scholia about a file without displaying
the file.)
\end{notate}

\subsubsection{Creating references} \label{creating-references}

\begin{notate}{Fancy references} \label{fancy-references}
In Note \ref{references} it isn't made completely clear what the set
of reference targets is comprised of.  One might be led to assume that
it is only possible to reference to an \emph{article}.  In the current
implementation, that assumption is correct; `follow-reference' calls
`display-article', which in turn only knows how to display
\emph{articles}.  However, in theory we could just as well reference
any object that the system knows how to display; so, if
`display-article' was overloaded to work with regions as well as
articles, for example, there would be no problem with referencing
regions (although the code for `make-reference-in-current-article'
would have to be adjusted slightly).  It would be nice to add this
feature, since HTML already has targets that are approximately this
general (well, more like ``go to a specific point within the given
document'').  It shouldn't be too much to ask to enable following a
reference and then performing a given action (e.g. ``find the first
occurance of \emph{foo} in the referenced document'').

Another relevant thing to point out is that we can have different
kinds of references appear in different colors (Note
\ref{color-and-style}); this is true of other sorts of linking as
well (Note \ref{generalized-links}).  Compare Note
\ref{thought-processes}.
\end{notate}

\begin{notate}{Abstract connections}
References have two \inp{about} fields, one comprising the reference's
target, and the other, the region from which the reference originates.
This shows how an abstract connection between two documents can be
formed by a third element, rather than such connections being the
exclusive domain of \inp{about} data itself.  This is an important
feature for semantic networks; larger-scale structures can be
expressed using basic pieces.
\end{notate}

\begin{notate}{On `genref'} \label{genref}
Instead of using `gensym' (which causes problems) we do something
similar to generate references with unique ids.  Note that this might
cause some problems when we go to the distributed way of doing things
(or even just save and restore articles with references in them after
making some intervening edits), since references with the same names
might mean different things.  But of course, this is true of all
scholia with the same names, so I propose not to worry about it too
much right now.

It may turn out to be advantageous to use a reference counter that is
local to each article.

Actually, the scheme proposed here seems pretty weak; saving files and
then reading them back in after Emacs has been shut down could cause
problems.  It might be much better to have a reference counter in each
metadata article, so that the collection of references associated with
a given article is always unique.  Furthermore, references associated
with a given article should possibly be recorded on that article's
metadata explicitly \emph{as references}.
\end{notate}

\begin{lisp}
(defvar *reference-counter* 0)

(defun genref ()
  (setq *reference-counter* (1+ *reference-counter*)))
\end{lisp}

\begin{notate}{On `make-reference-in-current-article'} \label{make-reference-in-current-article}
This function is similar to
`make-scholium-about-part-of-current-article' (Note
\ref{make-scholium-about-part-of-current-article}) except that the
\inp{type} is set to ``reference''.  References currently don't have any
\inp{text}, but we could later set things up to let them have a
docstring or something like that.

(Are backlinks working properly in an article that has several
references associated with it?)
\end{notate}

\begin{lisp}
(defun make-reference-in-current-article (beg end &optional target)
  (interactive "r")
  (let ((target (or target
                    (read-article-name))))
    (if name-of-current-article
        (when target
          (scholium `(reference ,name-of-current-article ,(genref))
                    nil
                    `((,name-of-current-article
                       (passage 
                        ,beg
                        ,end))
                      (,target))
                    'reference)
          (deactivate-mark))
      ;; Maybe the message should instead be an offer to make
      ;; a scholium about the current buffer?
      (message "Make some article current first."))))
\end{lisp}

\begin{notate}{Speedy reference creation}
It would be handy to be able to make a reference to an article and
automatically create and follow the reference, all at the same time.
(Maybe also subsets of these actions, like create but don't follow.)
Probably we could do something similar for generalized references.
\end{notate}

\begin{lisp}
(defun create-follow-reference ()
  (interactive)
  (let ((name (buffer-substring (point) (mark))))
    (unless (get-article name)
      (save-excursion
        (set-buffer (get-buffer-create name))
        (make-current-buffer-into-article name)))
    (make-reference-in-current-article (min (point) (mark))
                                       (max (point) (mark))
                                       name)
    (display-article name)))
\end{lisp}

\begin{notate}{Come-from references} \label{come-from-references}
As weird as it might sound, it could be handy to have a function for
adding references to the current article from some other article (or class
of articles).  See Note \ref{backlinks} for further comments.  (It
seems possible, though maybe not too likely, that we would be able to
use the outline of `make-reference-in-current-article' to create a generic
function for both ``come-from'' and ``goto'' references.)
\end{notate}

\begin{notate}{Inspecting or changing reference target, deleting reference, changing ``alt text''}
Various operations that you might like to do with a reference need should
to be supported.  changing the ``alt text'' should be supported
automatically -- just edit the text -- but there is the chance that
the user could screw things up and accidentally delete the whole
scholium if they do the wrong thing, so we may want to provide some
assistance.
\end{notate}

\begin{notate}{Reference component access} \label{reference-access-functions}
These functions give easy access to information specifying the
referenced article (the target), the region the reference applies to,
and the name of the article the reference lies in.  These functions
rely on the formulaic nature of the \inp{type} data of references,
namely, a link to the passage wherein the reference is made is given,
followed by a link to the referenced article.

Notice that these functions would have to change if we later
allow multiple sources (presumably, regions) to be part of the
same reference.
\end{notate}

\begin{lisp}
(defun reference-source-link (reference)
  (first (scholium-about reference)))

(defun reference-from-article (reference)
  (car (reference-source-link reference)))

(defun reference-to-article (reference)
 (car (second (scholium-about reference))))
\end{lisp}

\begin{notate}{Justification of the representation of references}
It might seem excessive to have backlinks to a reference on the
reference target and the reference source, but remember that we want
to be able to display references as scholia.  Now, one thing that
might be the case is that instead of storing a standard backlink on
the reference target, we might want to store a special
``referenceed-by'' backlink.  But \emph{that} is probably excessive,
at least, given our rendering scheme (see Note \ref{mark-things-up}).
\end{notate}

\subsubsection{New undirected articles} \label{new-undirected}

\begin{notate}{On `make-new-undirected-article'}
Although one could simply create a new buffer and add that buffer to
the article list (as in Section \ref{adding-buffers}), sometimes it
may be more intuitive to simply add a new undirected article directly
to the article list.
\end{notate}

\begin{lisp}
(defun make-new-undirected-article ()
  (interactive)
  (setq new-scholium-about nil)
  (make-scholium))
\end{lisp}

\subsubsection{Followups} \label{followups}

\begin{notate}{Introduction to followups}
This section is about making scholia about other already-existing
scholia.  It relies on markup features from Section \ref{rendering},
and its presence here is a bit ugly (Note \ref{principle-of-order}).
Perhaps it will be moved somewhere else later.
\end{notate}

\begin{notate}{Complex ``about'' features for scholia}
More complicated features like creating scholia directly about several
regions of a given scholium or like creating a scholium that relates
to several regions of several different scholia can be coded up soon,
following the style of section \ref{attached-main}.
\end{notate}

\begin{notate}{On `name-of-current-scholium'}
The function can be called from anywhere; ``current'' is defined
relative to the position of `point' in the Scholia Display buffer.
(It is natural to assume that there is only one current scholium,
given the way the contents of this buffer have been put together.)

Note that \emph{scholia} appear once in the Scholia Display buffer, so
there is only one ``current scholium'' (with one name) when we look at
things this way.  If we were going to do something like this for the
main article buffer, then we'd need to do a bit more.  (In fact,
various functions to associate marked regions with scholia need just
this sort of special touch.)
\end{notate}

\begin{lisp}
(defun name-of-current-scholium ()
  (interactive)
  (save-excursion
    (set-buffer (get-buffer-create "Scholia Display"))
    (let ((ret (car (scholia-named-at-point))))
      (if ret
          (message (format "%s" ret))
        (message "Position cursor on a scholium in Scholia Display."))
      ret)))

(defun make-scholium-about-current-scholium ()
  (interactive)
  (when (equal (buffer-name (current-buffer)) "Scholia Display")
    (let ((cur (name-of-current-scholium)))
      (when cur
        (progn (setq new-scholium-about `(((,cur))))
               (make-scholium))))))
\end{lisp}

\subsection{Further notes on adding articles to the digital library}

\begin{notate}{Forbid the creation of ``empty'' scholia?}
Perhaps `make-scholium' should disallow (or at least display an idiot
box about) the creation of ``empty'' scholia -- they seem somewhat
silly.
\end{notate}

\begin{notate}{Transcluded strings}
We'd should have some function to insert an auto-updating string (like
the name for the buffer as an article).  We would typically want such
auto-updating portions to appear in a specially colored face.
\end{notate}

\begin{notate}{Multi-reference creation?}
This section should probably cover: the event of interactively making
a scholium that relates to several different pieces of text from
several different articles.  Note that you could probably do things
like this by using the generic list, and marking things in different
ways that show their relationship together, the create the new
scholium from that list.
\end{notate}

\begin{notate}{Anonymous scholia}
Creating ``anonymous'' scholia should probably be made possible; by
which I mean, the names of these scholia would be filled in
automatically but would be non-significant (compare `gensym').  Of
course, if we could have automatic names that were significant that
would be cool too.
\end{notate}

\begin{notate}{Merging editorial comments}
It would be nice if there was an easy way to merge editorial comments
into the main document, corresponding to the ability to make such
comments in the first place (Note \ref{straightforward-applications}).
\end{notate}

\section{Rendering articles} \label{rendering}

\subsection{Formatting articles for display} \label{formatting-articles}

\begin{notate}{``Formatting'' versus ``display'' versus ``rendering''} \label{why-we-say-rendering}
The term ``formatting'' seems to capture a specific notion of
producing a user-visible version of an arbitrary article, that is then
``displayed''.  ``Rendering'' is meant to capture a more general idea,
which includes non-user-visable features (e.g. invisible text
properties).  Some sorts of articles will require further work to
``render'' (see Note \ref{transformative} and Note
\ref{functional}).
\end{notate}

\begin{notate}{Generality, or lack thereof, in the rendering system developed here}
It woulb be great to have all sorts of different kinds of rendering
systems.  (For one example additional to those mentioned in
\ref{why-we-say-rendering}, see Note
\ref{interactive-and-actionable-scholia}; we could supply other
examples!)  For now, we have limited ourselves to something pretty
basic.
\end{notate}

\begin{notate}{Producing plain text}
The function `sch-plain-text' uses various criteria to render scholium
system objects as a string (the collection of acceptable objects is
the same as the collection of link targets, plus articles specified
simply by name).  Its behavior can be fine-tuned by changing the
variable `sch-plain-text-hook'.
\end{notate}

\begin{notate}{On `sch-plain-text-hook'} \label{sch-plain-text-hook}
Alternative ways of setting the return value of `sch-plain-text'.
\end{notate}

\begin{lisp}
(defvar sch-plain-text-hook nil)
\end{lisp}

\begin{notate}{On `sch-plain-text'} \label{sch-plain-text}
This function is called by `mark-up-scholium' to render scholia that
are to be displayed alongside the main article, and also by
`transclude-article' (see Note \ref{transclude-article}) and other
functions in Section \ref{transclusion}.

Its goal is to turn arbitrary articles into strings.  This will be
done in different ways depending on the sort of article in question.
(And could be done in other ways depending on other stuff.)  Compare
Note \ref{user-crafted-semantics}.

Here's how it works: different kinds of ``objects'' are to be
distinguished from one another by simple tests -- is the `car' equal
to \verb|passage|?  Does looking up the object in the article table
produce anything?  Then act as appropriate, grabbing the text that is
specified.  Currently it works on input \emph{article names} or
input \emph{articles}.  (If input is both the name of an article
and an article, it will be treated as a name.)

Another thing that might be handy to be able to process is
\emph{simple strings}, which aren't article names, or articles
(obviously).

Note that links are rendered differently depending on context.

Links can't actually run from arbitrary object to arbitrary
object within the current version of the system.  That can
probably be fixed easily.

Converting the function to render scholium system objects in general
is a step towards answering the request in Note
\ref{name-of-current-article} about extending the purview of
`display-article'.  At that point, it will take more than just a
\inp{title} as argument, certainly; and it will need a ``triage''
phase in order to figure out what sort of object it has been applied
to; various modifications will have to be made so that it can be
applied to various sorts of objects.
\end{notate}

\begin{lisp}
(defun sch-plain-text (title-or-article)
  ;; this overloading of the input & reliance upon `get-article' to
  ;; sort things out... could probably be revised into something
  ;; better
  (let* ((obj (or (get-article title-or-article)
                  title-or-article))
         (text (scholium-text obj))
         (type (scholium-type obj))
         ret)
    ;; This seems like the correct way to override a `cond' form.
    (run-hooks 'sch-plain-text-hook)
    (when (not ret)
      (cond
       ((bufferp text)
        (save-excursion (set-buffer (get-buffer text))
                        (setq ret (buffer-string))))
       ((typedata-includes type 'file)
        (let ((bufs (buffer-list))
              (live nil))
          (while (and bufs (not live))
            (when (equal (buffer-file-name (car bufs))
                         (expand-file-name text))
              (setq live (car bufs)))
            (setq bufs (cdr bufs)))
          (if live
              (progn (set-buffer live)
                     (setq ret (buffer-string)))
            (find-file text)
            (setq ret (buffer-string))
            (kill-buffer (current-buffer)))))
       ;; these quoted things should presumably themselves be rendered
       ;; as links (and we probably don't need the crazy markup for
       ;; things about the whole buffer, ever)
       ((typedata-includes type 'reference)
        (if (equal (reference-to-article obj)
                   name-of-current-article)
            (setq ret
                  ;; it might be kind of cool to at least include a
                  ;; snippet of the context of this link, say 3 lines
                  ;; worth
                  (format "\"%s\" links here."
                          (reference-from-article obj)))
          (setq ret
                (format "This is a link from \"%s\" to \"%s\"."
                        (reference-from-article obj)
                        (reference-to-article obj)
                        obj))))
       ((stringp text)
        (setq ret text))
       (t
        (setq ret (format "%S" text)))))
    ret))
\end{lisp}

\begin{notate}{Turn list of ``links here'' items into listing}
It would be nice to be able to view all of the ``links here''
articles in a listing (as in Section \ref{catalog}), and from
there in a compilation (Note \ref{making-a-compilation}).
\end{notate}

\begin{notate}{Rendering special list forms}
With a small loss of transparency (note
\ref{principle-of-transparency}), we can set up some specially-encoded
\inp{text} fields.  See, e.g., Section \ref{transclusion}.  
\end{notate}

\begin{notate}{Strategy for `cond's}
Any time we see a `cond' in this code, we could wrap it in a backquote
and unquote a variable that says how to deal with things that come up
later in the code (see Note \ref{principle-of-order}).
\end{notate}

\begin{notate}{Actionable references} \label{actionable-references}
When displaying a reference as the main article, it would be nice to
have the names of the articles (or other objects) themselves rendered
as references to the articles (or other objects) in question.  These
wouldn't officially be ``references'' from the point of view of the
system (unless they were temporary ones), but they would act like
references for the user.

Some examples of this sort of thing appear in Section \ref{linear-browsing},
and are hooked in by `scholia-display-extras-hook' 
(Note \ref{scholia-display-extras-hook}).
\end{notate}

\begin{notate}{Render the names of articles?}
Depending on the value of a variable like `sch-use-names' or something
of that sort, this could print the names of the scholia above their
text contents.  We might also want to print a list of articles that
this article is ``about'' (assuming there is more than one).
\end{notate}

\subsection{Managing windows and buffers} \label{managing-windows-and-buffers}

\begin{notate}{Side-by-side buffers} \label{side-by-side-buffers}
The first display style shows the main article in one buffer and
scholia about this article in another buffer.  The scholia may be
attached to specific regions of the main article, in which case, this
connection is shown graphically.  Other display styles
(e.g. threading, see Note \ref{threading}) are in the works.
\end{notate}

\begin{notate}{Special treatment of buffers and files} \label{bias-towards-buffers}
Buffers and files need some special treatment for display, even if
they don't need to be given much attention at scholium-creation time
(Note \ref{scholium-creation-modalities}).  For example, it would be
nice to set up automatic display of scholia associated with a given
file if some spceial variable is set.  One class of buffers that
requires special treatment are the \emph{killed} ones (see Note
\ref{killed-buffers}).  This issue relates to Note
\ref{the-emacs-way}.
\end{notate}

\begin{notate}{Simpler side-by-side annotation}
A simpler method for doing side-by-side editing has been discussed
recently on the help-gnu-emacs mailing 
list\footnote{{\tt http://lists.gnu.org/archive/html/help-gnu-emacs/2005-07/msg00276.html}}.
This gives local features similar to some of those provided by the scholium
system, but none of the global features.  (We should make sure that we
actually \emph{can} simulate this behavior, I think it should be no problem.)
\end{notate}

\begin{notate}{A physical scholium system} \label{physical-scholium-system}
Working on this paper, I've taken to printing the document out
single-sided and binding the drafts along the margin, so that each
page has a blank facing page that I then use for notes.  Sounds
familar, doesn't it?  This approach is helping me understand how to
use paper better.  And it is giving me some ideas about useful
features to work into the electronic version.  See also Note
\ref{large-form}.
\end{notate}

\begin{notate}{Working with large-form scholium systems} \label{large-form}
With larger pieces of paper, or other media, it is possible to put
together scholium-based documents with more content and more than two
columns.  With a couple of screens, or a larger screen, useful things
along these lines could be accomplished with Emacs too.  (As it is,
two side-by-side windows is about all that can be managed on my
computer.)

Working with large format documents, I noticed some interesting
effects.  On the left, I might see a document assembled out of several
smaller articles (say, by identification), and on the right, I would
see all of the comments that applied to any of these articles, while a
third columns would give comments that applied to text in the other
columns, or to the document as a whole, or to any of the other
articles that had gone into building previous columns.

Even with just a couple of columns to work with, various complex
things like this should be possible to do on the computer (e.g. with
different colors or different levels of indentation).

(The point is, eventually should be able to make more interesting sort
compilations than those described in Section
\ref{quick-compilations}.)
\end{notate}

\begin{notate}{Avoiding information overload} \label{avoiding-information-overload}
A tricky option would be to just display the scholia that relate
to portions of the document that are on the current screen.  The paper
version of Note \ref{physical-scholium-system} does this by default.
But I'm not sure how to make Emacs behave this way.
\end{notate}

\begin{notate}{Context} \label{context}
Context is important.  This is one of the reasons that scholia
themselves are important -- they get comments situated in a certain
\emph{con}text.  The notion of context seems to be different for
\emph{text} and \emph{hypertext}.  In text, everthing appears in
context, whereas in hypertext, nearby things sometimes don't appear at
all; they reside on the other side of a link, and may never be seen by
a reader.  But this duality is somewhat misleading, in particular, it
is sometimes thecase that ``information overload'' would occur if
everything appeared on the same page or in the same place.  And
sometimes the context of the page can be misleading; regular text
frequently relies on global contextual features. (And of course,
readers might not actually see everything that is on the page!)  It is
an interesting feature of reality and human psychology that contexts
are often decipherable -- we can figure out how things fit together.
A ``good'' hypertext model should presumably include features that
reflect this aspect of our thinking, and allow users to explicitly
recontextualize things as they see fit.  For some technical
suggestions on this matter, see Note \ref{displaying-nearby-scholia}.
\end{notate}

\begin{notate}{Dot dot dot}
How should scholia in later generations be displayed?  In the default
mode, without threading, a ``dot dot dot'' would be useful.  (And in
the case of scholia that bridge the gap between two articles, the
``dot dot dot'' probably needs to have a special meaning.)  Anyway,
note that these dot-dot-dotted things could be found by a depth-first
search through the document collection.  (Find anything \inp{about}
the current article, then find anything \inp{about} that, etc., then
find the next thing \inp{about} the current article, 
etc.)\footnote{Compare the 2nd figure in Corneli and Krowne (2005)
(cf. Footnote \ref{corneli-krowne}, Page \pageref{corneli-krowne}).}.
% How to get page reference corresponding to a label?
\end{notate}

\begin{lisp}
(defvar pre-sch-win-config nil "Saved window configuration.")
(defvar sch-win-config nil "Saved window configuration.")

(defvar buffer-associated-with-current-article nil)
\end{lisp}

\begin{notate}{On `article-buffered'}
Here's a little convenience function that tells you whether the
article's text lives in a buffer or not.
\end{notate}

\begin{lisp}
(defun article-buffered (article)
  (bufferp (scholium-text article)))
\end{lisp}

\begin{notate}{Make new buffered article}
We should provide some command to make a new article in a new
buffer, all in one shot.
\end{notate}

\begin{notate}{Detaching articles from buffers}
Perhaps it should be possible to detach a buffered article from the
buffer that contains it.
\end{notate}

\begin{notate}{On `scholia-overwhelm-display'} \label{scholia-overwhelm-display}
This function displays scholium stuff.  It is called by
`display-article' (Note \ref{display-article}) and
`display-scholia-about-current-buffer'
(Note \ref{display-scholia-about-current-buffer}).  If
there is a main article buffer (i.e. the article to be displayed lives
in its own buffer), we use that buffer to display the article;
otherwise, we use the ``Main Article Display'' buffer.  Scholia that
match the appropriate set of conditions in `mark-things-up' will be
rendered to the ``Scholia Display'' buffer.
\end{notate}

\begin{lisp}
(defun scholia-overwhelm-display (text)
  (unless pre-sch-win-config
    (setq pre-sch-win-config (current-window-configuration)))
  (delete-other-windows)
  (split-window-horizontally)
  (if rendering-target-buffer
      (pop-to-buffer rendering-target-buffer t)
    (switch-to-buffer (get-buffer-create "Main Article Display") t))
  (erase-buffer)
  (insert text)
  (goto-char (point-min))
  (setq buffer-associated-with-current-article (current-buffer))
  (other-window 1)
  (switch-to-buffer (get-buffer-create "Scholia Display") t)
  (erase-buffer))
\end{lisp}

\begin{notate}{Switching between views} \label{switching-between-views}
We offer a few convenient functions for switching between the
article-plus-scholia browsing display and whatever came before.  It
would also be nice to offer a function for switching between the
article display and the generic listing display.
\end{notate}

\begin{lisp}
(defun back-to-normal ()
  (interactive)
  (setq sch-win-config (current-window-configuration))
  (set-window-configuration pre-sch-win-config)
  (setq pre-sch-win-config nil))

(defun back-to-other-view ()
  (interactive)
  (setq pre-sch-win-config (current-window-configuration))
  (set-window-configuration sch-win-config)
  (setq sch-win-config nil))
\end{lisp}

\subsection{Sorting scholia for markup purposes} \label{sorting}

\begin{notate}{Introduction to sorting scholia}
The main function in this section is `sort-scholia-by-beg-position',
which is called by `mark-things-up'.

The use for this function is to put scholia in the order in which they
will be rendered; the order is given by sorting the scholia on their
\inp{about} data.  Since there are several different kinds of
\inp{about} data, the sorting criteria are a little complicated.  For
one thing, each scholium may be about several different articles, but
here we only want to sort relative to their relationship to one given
article.

The criterion we use when sorting is: scholia that are about the whole
article go first; then, we rank any other scholia according to their
\emph{first} beginning position.  (Thus, any scholia that are about
several regions within the same article need to have a subsidiary sort
done to order the indicated regions; this is accomplished, as
needed, by the function `first-beginning-about-article', below.)
\end{notate}

\begin{notate}{On `first-beginning-about-article'}
Return 0 if \inp{about} is about all of \inp{article}, i.e., otherwise
return the character position of the first region within \inp{article}
that \inp{about} is actually about.  If there is none, return nil.
(Actually, let me note that in the usage we have established so far,
we would already know that in the case that \inp{about} is a string,
it is about the article here, and in the case that \inp{about} is a
one-layer list, similarly; so some of the tests we do here are as yet
unneeded.)

Here we're assuming that if there is \emph{some} link to the article
that isn't a passage link, then we treat the link as about the article
as a whole.  The case in which the link is about the whole article
\emph{and} some part of the article might possibly be better treated
some other way; but I'll leave that case for subsequent work.
\end{notate}

\begin{lisp}
(defun first-beginning-about-article (about article)
  (cond
   ;; condition for the scholium to be about the article as a whole.
   ;; The condition is that there is _some_ link to the article that
   ;; is NOT a ``passage''-type link.  In this case, we just return 0.
   ((member-if (lambda (link) 
                 (and (equal (car link) article)
                      (not (link-type-accessor link 'passage))))
               about)
    0)
   ;; else, collect the regions of `article' that `about' indicates,
   ;; and sort them.
   (t
    (let* ((marking-links
            (let (marked)
              (mapc (lambda (elt)
                      (when (and
                             (typedata-includes-passage (link-type elt))
                             (equal (linked-to-article elt) article))
                        (setq marked (cons elt marked))))
                    about)
              marked))
           (earliest-link
            (car (sort
                  marking-links
                  (lambda (link1 link2)
                    (< (link-beginning link1)
                       (link-beginning link2)))))))
      (link-beginning earliest-link)))))
\end{lisp}

\begin{notate}{On `sort-scholia-by-beg-position'} \label{sort-scholia-by-beg-position}
This function orders \inp{scholia} according to the magnitude of the
smallest beginning a region of \inp{article} that the input scholia
mark.  All links are considered when finding the first marked region.
\end{notate}

\begin{lisp}
(defun sort-scholia-by-beg-position (scholia article)
  (setq
   scholia
   (sort scholia
         (lambda (scholium1 scholium2)
           (let ((beg1 (first-beginning-about-article 
                        (scholium-about scholium1)
                        article))
                 (beg2 (first-beginning-about-article
                        (scholium-about scholium2)
                        article)))
             (and beg1
                  beg2
                  (< beg1 beg2)))))))
\end{lisp}

\begin{notate}{Incremental markup} \label{incremental-markup}
Instead of sorting everything each time we display, it would be nice
to handle things incrementally: just add new markup for the newly
created scholium, and store things in their natural order after
reparsing.  I think that this approach should work, but it would
involve making a number of changes to the code.  (Note that
in the setup we're using, updates to metadata may have to be
wholesale, even if updates to text properties are incremental.)
\end{notate}

\subsection{Marking things up} \label{marking-things-up}

\begin{notate}{Introduction to marking things up}
As mentioned in the introduction (Note
\ref{text-properties-and-overlays}): \emph{when articles are
  displayed, both overlays and text properties are used for markup}.
Overlays make it possible to put visible markup into font locked
buffers.  However, when text is cut and pasted or whatever, overlays
disappear, so we use text properties too, to add a needed degree of
permanence to the markup.
\end{notate}

\subsubsection{Faces for overlays}

\begin{notate}{A more abstract face function}
Perhaps `new-simple-scholium-face' could take an argument -- say the
scholium -- and provide a different face for each user, or a different
face for each type, or whatever.  Probably there should be a variable
called `scholium-face-function' that points you to the \emph{actual}
function that is going to be in use.  Using this sort of abstraction
throughout would certainly facilitate user customization, and so,
would be a good thing.
\end{notate}

\begin{notate}{On `new-simple-scholium-face'}
Adapted from ttn's {\tt ricette-mode.el}. \emph{Nota bene}: new faces
are automatically customizable, so if you don't like the way they
look, you can change them.
\end{notate}

\begin{lisp}
(defmacro new-simple-scholium-face (num color doc)
  (let ((prop (intern (concat "sch-face-" (int-to-string num)))))
    `(progn
       (defvar  ,prop ',prop)
       (defface ,prop
         '((t (:foreground ,(symbol-name color)
               :underline ,(symbol-name color)))) ,doc))))
\end{lisp}

\begin{notate}{Underlining versus foreground} \label{underlining-versus-foreground}
It is worth considering using the underline attribute instead or in
addition to the foreground attribute: underlining would be less
obtrusive in documents that already use faces.  We could have two
variants; one with underlining for the main article, one foreground
for the scholia display.  We could also give users some
immediately-customizable options.  (Do the default colors I picked out
work well?)
\end{notate}

\begin{lisp}
(new-simple-scholium-face 1 maroon1 "First scholium face.")
(new-simple-scholium-face 2 aquamarine1 "Second scholium face.")
(new-simple-scholium-face 3 IndianRed1 "Third scholium face.")
(new-simple-scholium-face 4 yellow1 "Fourth scholium face.")
(new-simple-scholium-face 5 firebrick1 "Fifth scholium face.")
(new-simple-scholium-face 6 plum1 "Sixth scholium face.")
\end{lisp}

\begin{notate}{Reference face} \label{reference-face}
A special face for references.  We don't currently have a special face
for \emph{visited} references, but this can be added if/when we start
keeping track of which references have been visited (see Note
\ref{visited-references}).  We might also want to record and display
information about visited articles in general.
\end{notate}

\begin{lisp}
(defface sch-reference-face
  '((t (:foreground "red" :underline "red")))
  "Face for references in the scholium system.")
\end{lisp}

\begin{notate}{Special face for scholia about whole article?}
It might be good to have a special face (and perhaps other
special treatment, like a special section of the scholia display
buffer) for scholia that apply to the whole of the main article.
\end{notate}

\begin{notate}{Special face for transclusions that have been collapsed} \label{collapsed-face}
We're going to have to have a special face for transclusions that
have been collapsed (see Note \ref{transcluding-and-outlines}).
\end{notate}

\subsubsection{Masks and miscellaneous markup mechanics}

\begin{notate}{Color by number} \label{color-by-number}
The `scholia-count' variable keeps track of how many scholia have been
 displayed.  The `scholium-face' function selects a face to use when
 displaying the next scholium according to this count.  This simple
 display mechanism seems sort of lame (hence, it is turned on with the
 `use-crazy-font-lock' variable); better things may come later, see,
 e.g. Note \ref{color-and-style}.

Note that turning off ``crazy font lock'' makes it so that no text
properties are added to the buffer, but of course it would be best if
text properties were added and overlays omitted.  On the other hand,
we need to do some thinking to sort out the use of text properties
versus the use of overlays.  Cutting and pasting multiple copies of
some markup in the same buffer may cause some trouble at commit time.
See Section \ref{transclusion}; also, compare Note
\ref{aggressive-dop}.
\end{notate}

\begin{lisp}
(defvar scholia-count 0 "Number of scholia about the article found.")

(defun scholium-face ()
  (let ((short-count (mod scholia-count 6)))
    (cond ((eq short-count 0)
           'sch-face-1)
          ((eq short-count 1)
           'sch-face-2)
          ((eq short-count 2)
           'sch-face-3)
          ((eq short-count 3)
           'sch-face-4)
          ((eq short-count 4)
           'sch-face-5)
          ((eq short-count 5)
           'sch-face-6))))

(defvar use-crazy-font-lock t)
(defvar main-article-overlays nil)
(defvar scholia-overlays nil)
\end{lisp}

\begin{notate}{Overlays and text properties adjust when editing}
By default, overlays and text properties will move appropriately when
text is added to the buffer.  This is important!
\end{notate}

\begin{notate}{Access to rendering target buffer} \label{access-to-rendering-target-buffer}
The rendering target buffer is either ``Main Article Display'' if the
current article isn't buffered, or whatever buffer the article lives
in if it is buffered.  (Well, actually, we can always redirect
rendering to any buffer of our choice; but the preceding statement
accurately describes the default operation.)  We use
`get-rendering-target-buffer' as a shorthand when we grab the
rendering target buffer, and `set-buffer-to-rendering-target-buffer'
to make that buffer current for editing.  (Note that we \emph{could}
go about this in a slightly different way, namely set the
`rendering-target-buffer' variable to ``Main Article Display'' instead
of `nil' when there is nothing to override the default -- but what
would be the point?)
\end{notate}

\begin{lisp}
(defun get-rendering-target-buffer ()
  (get-buffer (or rendering-target-buffer
                  (get-buffer "Main Article Display"))))

(defun set-buffer-to-rendering-target-buffer ()
  (set-buffer (get-rendering-target-buffer)))
\end{lisp}

\begin{notate}{The `scholia' property} \label{the-scholia-property}
At render time, we use a text property (which we call
\emph{`scholia'}) to associate regions with the scholia that are
attached to them.  A region can be associated with more than one
scholium, \emph{so the `scholia' text property stores a list}.
Specifically, a list identifying the \emph{links} that associate
various scholia with this region (see Note \ref{link-ids}).

This information helps us restore order to shifted markup when
committing edits (see Section \ref{parsing}).
\end{notate}

\begin{notate}{Link-id's} \label{link-ids}
The elements of stored in the `scholia' text property's list are
ordered pairs (two-element lists), each containing information of the
following form: (1) the \emph{name} of a scholium associated with this
buffer position; paired with, (2), the link number corresponding to
(one of) the link(s) through which said scholium came to be associated
with the particular region being marked up.  

Lists of this form are a fairly common idiom in this work, and they
get a special name, the \emph{link-id}.
\end{notate}

\begin{notate}{Experiment with many `scholium' properties} \label{many-scholium-properties}
All else equal, it might be advantageous to use independent `scholia'
properties instead of just one `scholium' property (see Note
\ref{inserting-text-in-middle-of-marked-up-region}).

Accordingly, I tried switching over to using one `scholium' property
for each marked-up region.  These properties were given somewhat
complicated names -- namely, the link-ids that are currently stored as
elements of the `scholia' property (Note \ref{the-scholia-property}).
However, this doesn't work, as suggested by the following example.
\begin{idea}
(progn (put-text-property (point) (1+ (point)) '(foo 1) t)
       (put-text-property (point) (1+ (point)) '(foo 1) nil)
       (text-properties-at (point)))
\end{idea}
This is because\footnote{Thanks Andreas Schwab, help-gnu-emacs,
2005/12/12. \label{so-called-text-property-bug}}
\begin{idea}
(eq '(foo 1) '(foo 1)) ;=> nil
\end{idea}

The Emacs text property engine is uniformly `eq'-based instead of
`equal'-based.  (So, if we happened to have an association between
link-id's and integers, this would have worked, but that seems like a
silly kludge.)

Indeed, this `eq' preference is pretty much fatal to the utility of
non-symbol text properties (unless we managed to hang directly onto
the actual link-ids that would be used for the names of the text
properties, which seems infeasible; or alternatively used some other
sort of weird work-around, as above).  A further difficulty is
associated with the fact that these various `scholium' text properties
would be indiscriminately mixed in with any other text properties that
happened to be stored at point, requiring filtering for any useful en
masse processing. 

I have some ideas that could possibly improve the prospects for doing
away with the `scholia' property and replacing it with several
`scholium' properties, but it seems like anything realistic would
hacking Emacs C.  But since it seems that there are no immediate
problems associated with using just one `scholia' property, we plunge
ahead that way.
\end{notate}

\begin{notate}{On `add-to-scholia-property-within-region'} \label{add-to-scholia-property-within-region}
For each character in the region between \inp{beg} and \inp{end}, this
function grabs the `scholia' property and replaces it with a version
that has been modified to include the input \inp{value} (a link-id;
see Note \ref{the-scholia-property}).

The same property is used in the Scholia Display buffer, although that
may be a bit of an abuse (cf. Note
\ref{using-scholia-property-for-everything}); something simpler would
work for our needs there -- but the current way is expedient.  Also,
it seems to be suggestive -- perhaps in the future we'll be able to
treat the Scholia Display buffer as a proper scholium-based article
itself, presumably by using transclusion and identification, as
appropriate (see Section \ref{derivative}).
\end{notate}

\begin{lisp}
(defun add-to-scholia-property-within-region (start end value)
  (while (< start end)
    (put-text-property start (1+ start) 'scholia
                       (add-to-or-start-list
                        (get-text-property start 'scholia)
                        value))
    (setq start (1+ start))))
\end{lisp}

\begin{notate}{On `current-markup'} \label{current-markup}
This variable will be used to record the regions of the article being
displayed which have scholia attached to them and are, consequently,
marked up.  This information will be stored at markup time by
`mark-up-region' (Note \ref{mark-up-region}).  It is important to have
this information on record so that we have something to compare to
after editing takes place (see Section \ref{committing}).

Its format is a list of elements of the form
\begin{idea}
((<name> <link number>) <beg> <end>)
\end{idea}
where `beg' and `end' denote the beginning and end of the region
marked up via the specified link (but see Note 
\ref{investigating-current-markup}!).
\end{notate}

\begin{lisp}
(defvar current-markup nil)
\end{lisp}

\begin{notate}{Investigating `current-markup' in the context of masks} \label{investigating-current-markup}
If the link in question is being masked, then the format of
`current-markup' changes from the form described in Note
\ref{current-markup} to the following:
\begin{idea}
((mask (<name> <link number>)) <beg> <end>)
\end{idea}
where `beg' and `end' denote the beginning and ending of
one particular region coming from the mask for the indicated
link.

I'm not sure this is really sufficient information.  Should we know
\emph{which part} of the mask we're dealing with?  I.e., use something
like a link-id, but for mask components?
\end{notate}

\begin{notate}{On `mark-up-region'} \label{mark-up-region}
Like `add-to-scholia-property-within-region', but also adds to
`current-markup' (see Note \ref{current-markup}).
\end{notate}

\begin{lisp}
(defun mark-up-region (start end value)
  (add-to-scholia-property-within-region start end value)
  (setq current-markup
        (add-to-or-start-list current-markup
                              (list value start end))))
\end{lisp}

\begin{notate}{On `non-printing-types'}
This variable will record a list of the types of scholia that we
typically don't want to print when displaying an article.  The
`derives-from' type is an example of a scholium that we typically
don't want to print; see Note \ref{include-article}.

User should temporarily set `non-printing-types' to `nil' before
`mark-things-up' runs whenever they wish to display all scholia.
\end{notate}

\begin{lisp}
(defvar non-printing-types nil)
\end{lisp}

\begin{notate}{Masking links} \label{masking-links}
It is sometimes necessary to use links that point at something other
than the thing that they were pointed at originally -- or more
typically, the same thing, but found in a different place (Note
\ref{updating-link-extent}).

For example, consider the common case of a link pointing at some
region of text found in a given article.  If the linked-to article
gets edited, the link under consideration may have to be adjusted if
it is going to continue to point at the ``correct'' region.

Thus, when a linked-to article changes, it will typically have to
communicate changes to back to the linking article; or rather, to the
metadata article associated with the linking article (Note
\ref{metadata-article-overview}).  These changes take the form of
\emph{masks}, which cover and re-route links.  

A given mask may continue to change as the linked-to article changes.
The strategy for keeping masks up-to-date is as follows.  When we
discover (at commit time) that a link needs to be redirected, an entry
in the ``masks'' metadata field of the article containing the link is
created or adjusted.  Specifically, the new link(s) designed to
replace the old link are stored as an entry on the ``masks'' list, as
an ordered pair consisting of: (1) old link-id; followed by, (2) the
new link(s) replacing the old one.  Subsequent changes in the
linked-to article cause old masks to be deleted and new masks to be
swapped in; we never do ``recursive masking''.

Mask-bearing links are clearly distinguished as such (at markup time),
so that we can easily identify them later (when parsing markup).  The
function that generates the mask-identifying tags is
`compute-usable-about-data' (Note \ref{compute-usable-about-data}),
and the function that picks these tags out from among the others at
commit time is `store-link-masks' (Note \ref{store-link-masks}).
Indeed, `store-link-masks' is responsible for creating new masks as
well.

At the other end of the process, the function
`compute-usable-about-data' examines each link from the \inp{about}
data of the linking article together with any corresponding masks, and
replaces any link that is masked with the data coming from that mask
(Note \ref{compute-usable-about-data}).

(Changes to the linking article's \inp{about} data can change
link-id's, or do away with the requirement for a particular mask
altogether -- this will have to be dealt with when we have mechanisms
for changing \inp{about} data!)
\end{notate}

\begin{notate}{Masks} \label{masks}
In light of the comments in Note \ref{masking-links}, it is possible
to think of a mask as a map that takes a given link to to a region,
and replaces it with a list of several regions.

The format of a mask, then, is
\begin{idea}
((<name> <link number>) &rest regions)
\end{idea}
where the regions are pairs (two-element lists), each giving the
beginning and end of a region that the link is being redirected to.

This format lacks generality!  Certainly a link that is pointing to
something other than a region may need to be redirected, for example,
a link pointing at a page, when the page gets deleted.  For the time
being, these interesting cases can be dealt with through \emph{ad hoc}
measures.  Eventually we'll want to make a study of all of the
different redirection cases; but getting the platform working
reasonably well takes priority.  (See also Note
\ref{generality-of-masks}.)

Finally, observe that the format used here is related to the format of
the `current-markup' variable (Note \ref{current-markup}).

For an interesting meditation on masking perceptions, see
Lem\footnote{Stanislaw Lem, The Futurological Congress}.
\end{notate}

\begin{notate}{Generality of masks} \label{generality-of-masks}
Once we have more general sorts of masks (Note \ref{masks}), we'll
have a host of new and interesting conditions to consider.  Different
types of links will presumably have to be masked in different ways,
and under different conditions.  

Furthermore, we will eventually be masking \inp{text} data as well as
\inp{about} data.
\end{notate}

\begin{notate}{On `compute-usable-about-data'} \label{compute-usable-about-data}
This function runs within the scope of `mark-things-up' (Note
\ref{mark-things-up}).  Its role is to identify the regions to be
marked up, based on the \inp{about} data expressed by attached
scholia, and augmented by any masks associated with this data.  In
short, this is the mask-applying part of the algorithm described in
Note \ref{masking-links}.

The way it works is as follows.  We look through all of the links
(\emph{although we really don't need to do this anymore, given that we
  have specifically identified the relevant links by using link-id'ed
  backlinks in the article being displayed; the change would have to
  come in at the level of `mark-things-up' or higher}) -- and if the
link is relevant, then it will be reflected in the value returned by
`compute-usable-about-data'.  However, if the link is masked, it will
be the mask that is reflected, instead of the link itself.

Recall that we can't `assoc' the link across the masks; we need
to `assoc' the link-id instead.

The format of the return value is kind of ugly; we could almost
certainly do without the extra `list' layer.

Note that in the return value, the name of the \emph{scholium} is
supplied, not the name of the linked-to article -- we'll subsequently
know (when applying markup) that all of the links apply to the
linked-to article, whereas we won't know where these links are coming
from unless we record that specially.
\end{notate}

\begin{lisp}
(defun compute-usable-about-data ()
  (let (usable-data
        (about (scholium-about scholium))
        (masks (get-metadata-field 'masks
                                   (scholium-name
                                    scholium)))
        (link-number 0))
    (dolist (link about)
      (setq link-number (1+ link-number))
      (when (link-about-article-p link name-of-current-article)
        (let* ((link-id (list (scholium-name scholium) link-number))
               (mask (assoc link-id masks)))
          (if mask
              (dolist (reg (cdr mask))
                (setq usable-data
                      (add-to-or-start-list
                       usable-data
                       `((mask ,(car mask)) ,@reg))))
            (setq usable-data
                  (add-to-or-start-list
                   usable-data
                   (append (list (list (scholium-name scholium)
                                       link-number))
                           (let ((beg (link-beginning link)))
                             (if beg 
                                 (list beg (link-end link))
                               (list nil))))))))))
    usable-data))
\end{lisp}

\begin{notate}{On `mark-things-up-customizations'} \label{mark-things-up-customizations}
This may not have to be used.
\end{notate}

\begin{lisp}
(defvar mark-things-up-customizations nil)
\end{lisp}

\begin{notate}{On `scholia-display-pre-update-hook'} \label{scholia-display-pre-update-hook}
This gives us a chance to do various customizations to the environment
before the main part of `mark-things-up' runs.

I imagine that the role that it will play will be similar to the one
played by the functions that run right in `pre-mark-up', namely to zap
variables and settings that we don't want to have around anymore.
\end{notate}

\begin{lisp}
(defvar scholia-display-pre-update-hook nil)
\end{lisp}

\begin{notate}{On `pre-mark-up' }
This function generally zaps things, to prepare for markup.
\end{notate}

\begin{lisp}
(defun pre-mark-up ()
  (setq current-markup nil)
  (save-excursion
    (set-buffer-to-rendering-target-buffer)
    (remove-list-of-text-properties (point-min)
                                    (point-max)
                                    '(scholia)))
  (mapcar #'delete-overlay main-article-overlays)
  (setq main-article-overlays nil)
  (setq scholia-count 0))
\end{lisp}

\begin{notate}{On `mark-things-up-hook'} \label{mark-things-up-hook}
For unconditional customizations to `mark-things-up'.
The functions added to this hook run after everything
else `mark-things-up' does is done.
\end{notate}

\begin{lisp}
(defvar mark-things-up-hook nil)
\end{lisp}

\begin{notate}{On `mark-things-up'} \label{mark-things-up}
This function assembles text, text properties, and overlays: it is the
main rendering engine, called by `display-article' to get
scholia and appropriate markup onto the screen.  It is important
that `raw-scholia' be defined (and be a list of scholia)
for this function to work properly.

The first part of the plot is to delete the old markup; we call
`pre-mark-up' to take care of this.

A list of ``raw scholia'' is expected to be present in the context in
which this function runs; typically this data is provided by
`display-article' (see Note \ref{display-article}), but it can be
supplied by any stand-in (e.g. `display-scholia-about-current-buffer'
of Note \ref{display-scholia-about-current-buffer}).

The raw scholia are first sorted using `sort-scholia-by-beg-position'
(Note \ref{sort-scholia-by-beg-position}) and then translated into a
usable form by `compute-usable-about-data' (Note
\ref{compute-usable-about-data}).

The function `mark-things-up' can handle different sorts of scholia
differently (e.g., references are marked up in the conventional way);
see Note \ref{ancillary-mark-things-up}.

In order to \emph{selectively} display scholia, the `raw-scholia'
variable should be modified before this function runs.
\end{notate}

\begin{lisp}
(defun mark-things-up ()
  (pre-mark-up)
  (let ((scholia (sort-scholia-by-beg-position
                  raw-scholia
                  name-of-current-article)))
    (dolist (scholium scholia)
      (unless (typedata-includes-element-of-list
               (scholium-type scholium)
               non-printing-types)
        (let ((usable-about-data (compute-usable-about-data))
              (current-position-in-scholia-display (point)))
          (cond
           ((and
             (typedata-includes (scholium-type scholium) 'reference)
             (equal (reference-from-article scholium)
                    name-of-current-article))
            (mark-up-reference))
           (t
            (mark-up-scholium)))))))
  (run-hooks 'mark-things-up-hook))
\end{lisp}

\begin{notate}{Ancillary functions for `mark-things-up'} \label{ancillary-mark-things-up}
The functions that assemble and apply the markup, and in particular
 the value stored on the `scholia' property, run within the scope of
 `mark-things-up'.  The `scholia' property is built according to
the description from Note \ref{the-scholia-property}.

References are rendered by `mark-up-reference'; regular
scholia are rendered by `mark-up-scholium'.  \emph{Note that there are
  only two different nontrivial kinds of markup at present.}

Some \emph{very} ancillary functions are described in Note
\ref{functions-for-adding-overlays}.
\end{notate}

\begin{notate}{Functions for adding overlays} \label{functions-for-adding-overlays}
These function run within the scope of `mark-up-scholium' to
add overlays to the display.
\end{notate}

\begin{lisp}
(defun add-overlays-in-scholia-display-buffer ()
  (setq scholia-overlays
        (cons
         (make-overlay current-position-in-scholia-display
                       (point)
                       (get-buffer "Scholia Display")
                       t)
         scholia-overlays))
  (overlay-put (car scholia-overlays)
               'face (scholium-face)))

(defun add-overlays-in-rendering-target-buffer ()
  (setq main-article-overlays
        (cons
         (make-overlay (second elt)
                       (third elt)
                       (get-rendering-target-buffer)
                       t)
         main-article-overlays))
  (overlay-put (car main-article-overlays)
               'face (scholium-face)))
\end{lisp}

\begin{notate}{On `mark-up-reference'} \label{mark-up-reference}
We call this function from `mark-things-up' (Note
\ref{mark-things-up}) to render a scholium if it has reference type
and its \inp{about} data indicates that that the reference originates
from (i.e., appears in) the current article.  See Note
\ref{reference-access-functions} for a description of the format of
the return value of the function `reference-source-link' used here.

References should perhaps be rendered differently depending on their
sub-type (in particular, a different face could be used for references
with different sub-types).

(Do we really want to loop through all of the elements of
`usable-about-data'?  Mightn't there be something there corresponding
to the linked-to article?)
\end{notate}

\begin{lisp}
(defun mark-up-reference ()
  (save-excursion
    (set-buffer-to-rendering-target-buffer)
    (dolist (elt usable-about-data)
      (mark-up-region (second elt)
                      (third elt)
                      (first elt))
      (when use-crazy-font-lock
        (setq main-article-overlays
              (cons
               (make-overlay (second elt)
                             (third elt)
                             (get-rendering-target-buffer)
                             t)
               main-article-overlays))
        (overlay-put (car main-article-overlays)
                     'face 'sch-reference-face)))))
\end{lisp}

\begin{notate}{Using the `scholia' property for everything!} \label{using-scholia-property-for-everything}
Notice that we use the same `scholia' property for references, and not
some special property.  This seems to make reparsing easier (see
Section \ref{parsing}) -- but we can also easily filter the
references out of the collection of all scholia when needed (Note
\ref{follow-reference}).
\end{notate}

\begin{notate}{Alternate reference display}
In Note \ref{references} we asserted that references will be displayed
as markup.  However, we could certainly set things up so that we could
switch between the normal view and a view where references are
displayed as ``proper scholia''.
\end{notate}

\begin{notate}{On `mark-up-scholium'}
This inserts scholia and marks them up, together with the marked up
regions (when these exist; the function does both whole-article
scholia and region-specific scholia).

The `scholia-count' variable is set for purposes of face selection;
see Note \ref{color-by-number}.
\end{notate}

\begin{lisp}
(defun mark-up-scholium ()
  ;; this part takes place in the scholia display buffer
  (insert (sch-plain-text (scholium-name scholium)))
  (add-to-scholia-property-within-region
   current-position-in-scholia-display
   (point)
   ;; add a list to make it possible to reuse the scholium property
   (list (scholium-name scholium)))
  (when use-crazy-font-lock
    (add-overlays-in-scholia-display-buffer))
  (insert "\n\n")
  ;; this part is relevant to the buffer containing the main article
  (dolist (elt usable-about-data)
    (when (second elt)
      (save-excursion
        (set-buffer-to-rendering-target-buffer)
        (mark-up-region (second elt)
                        (third elt)
                        (first elt))
        (when use-crazy-font-lock
          (add-overlays-in-rendering-target-buffer)))))
  ;; adjust count once everything else is done here, so same count
  ;; applies in both buffers (useful for coloration purposes)
  (setq scholia-count (1+ scholia-count)))
\end{lisp}

\begin{notate}{Markup in the Scholia Display buffer} \label{markup-in-scholia-buffer}
There is no technical reason to use overlays here, because we assume
that this buffer is not subject to font-lock (i.e. we could use text
properties for everything).  But, for the sake of uniformity, we
always use overlays for fontification.  Also, there's no particular
reason to maintain the `scholia' property as a list in this buffer, but
we do that too.
\end{notate}

\begin{notate}{Make ``Scholia Display'' read-only?}
Maybe `mark-things-up' should set and unset read-only status for the
Scholia Display buffer.  Some of the functions we've written so far do
toggle read-only status.  We should make a firm decision one way or
the other!

At present, it seems to me that it is best to have it be editable and
have code ready to propagate changes back to their sources.  Indeed,
this would make a good example of the power of this system.  See Note
\ref{precedent-for-identification}.
\end{notate}

\subsubsection{Further notes on marking things up}

\begin{notate}{Highlighting the current scholium}
Note that it may prove to be useful to highlight the \emph{current}
scholium in some way, or current \emph{scholia} with some kind of
color-coded depth showing how many scholia are about the current piece
of text.  (``Current'' here comes from where the point is in the main
article display, but I suppose it might be reasonable to do something
similar with the scholia display buffer.)
\end{notate}

\begin{notate}{Semantics of color}
The actual markup behavior should depend on the semantics that are
currently \emph{en vogue}, see Note \ref{color-and-style} and Note
\ref{user-crafted-semantics}.  Maybe we should be using a hook or a
redefinable function to handle the ``when use-crazy-font-lock'' case.
\end{notate}

\begin{notate}{Lots of different overlay styles possible} \label{overlay-styles}
In theory we could have a lot of different overlay styles.
Implementing these styles seems a little tricky, but isn't too
impossible.  Note that some of the different overlay setups may
benefit from having more information than just the name stored locally
(assuming that it takes a lot of time to look things up by name to
find more properties, which isn't necessarily a fair assumption in the
case of hash tables).

One example of something that would be nice would be to have nested
scholia display well.  (I'm sure they would even work consistently at
present.)  It may be that delimiters would be a more effective way of
illustrating such relationships.
\end{notate}

\begin{notate}{Sometimes we don't want overlays at all}
Another point is that sometimes we don't want overlays to show up at
all -- they could be very distracting when attached to code, for
example.  Whether or not an overlay is shown in a given instance may
depend on global state variables, properties of the article that is
being marked up, or specific data that has been stored as part of a
scholium.  (We'll have to add more code to handle these sorts of
criteria.)
\end{notate}

\begin{notate}{Emacs task: blinking?} \label{blinking}
This is a pretty minor issue, but it would be cool to hack a
\emph{display independent} blink property into emacs.  Blinking
already works for cursors.  So, how to do it for text?  The point is
that if there was a ``blink'' feature in Emacs, the ``move to region''
things could make the text they find blink (this would work for
monochrome displays of Note \ref{monochrome} just as well as any
other).
\end{notate}

\begin{notate}{Displaying nearby scholia} \label{displaying-nearby-scholia}
A nice display system will put some of the scholia nearby the current
document onscreen.  The default operation of `mark-things-up'
(performed through the offices of `sort-scholia-by-beg-position') is
to show an article together with \emph{its} scholia in order; Note
\ref{ancestors} talks about going the other way.  But we can do still
more complicated things.

People like having nearby things to look at.  This is one reason that
print dictionaries, for example, are kind of fun.  You look at the
dictionary, and you can see words that are nearby lexicographically;
sometimes they are related and sometimes they aren't.

In different contexts, different sorts of ``closeness'' may be
relevant, and different sorts semantics would help make the contextual
display useful.
\end{notate}

\subsection{Display interface} \label{display-interface}

\begin{notate}{Selective Displays} \label{selective-displays}
More selective displays could be offered.  For example, we might
display only the articles that are ``about'' the current article
\emph{and} that are of some specified ``type''.  We could also take
bookkeeping information into account, or ``meta-level'' information
like what subcollection (or namespace?) of the digital library the
article is located in.  Metalevel information (maybe by definition)
comes from outside of the system itself, for example, we might display
only scholia from a certain directory.  These sorts of deployments are
where the real power of this system lies.  (See Note
\ref{selection-methods} for general comments along these lines, and
Note \ref{ancestors} for somewhat similar thoughts along a diverging
line.)
\end{notate}

\subsubsection{Different treatment for buffered and non-buffered articles}

\begin{notate}{On `rendering-target-buffer'} \label{rendering-target-buffer}
If a given \inp{name} is associated with a buffered article, then that
buffer will be where the article is displayed.  If the article to be
displayed is in a buffer, this variable will be set that buffer
object.  Otherwise it will be set to nil.
\end{notate}

\begin{lisp}
(defvar rendering-target-buffer nil)
\end{lisp}

\begin{notate}{Dealing with killed buffers} \label{killed-buffers}
For some reason, `rendering-target-buffer' gets set to the ``killed
buffer'' object when the buffer containing the main article is killed.
That isn't nice; presumably we'll need to add a function to the
`kill-buffer-hook' that will ensure that if this buffer is killed,
this variable will be set back to nil (or whatever it should in fact
be set to).  We will presumably also have to remove this function from
the hook when the main article changes.
\end{notate}

\begin{notate}{Better way to name (or access) buffered articles} \label{name-or-access-buffered-articles}
Either we should search on the \inp{text} field for the current
buffer, and ignore the name, or we should store the ``official'' name
of the buffer as a buffer-local variable.  (E.g. this buffer
\emph{should} be called ``SBDM for CBPP'' but
`display-scholia-about-current-buffer' can, at present, only access
the buffer by its `buffer-name'.  I think that adding a buffer-local
variable at creation time would probably be very straightforward, then
we just need to look-up that variable instead of using `buffer-name'.
(Of course, we could make `buffer-name' be the default for the
``buffer article name''.)  Note that unless the article is created by
the approved means, we won't have a chance to store the variable; this
suggests that the display section should come after the creation
section (which makes plenty of sense).  Note that \emph{everywhere}
`buffer-name' appears in this paper things will have to be modified if
we make this sort of switch.
\end{notate}

\subsubsection{Display mechanics}

\begin{notate}{What just happened to the screen?}
The simple display mechanism we use here will overwhelm your whole
display (see Note \ref{scholia-overwhelm-display}); use {\tt M-x
  back-to-normal} to return to the pre-display window configuration.
\end{notate}

\begin{notate}{Making `sch-plain-text' work with weird types}
We should either make sure that `sch-plain-text' can handle all
articles with many different \inp{type}s.
\end{notate}

\begin{notate}{On `scholia-display-post-update-hook'}
This provides a way to customize the article (and
scholia) being displayed.
\end{notate}

\begin{lisp}
(defvar scholia-display-post-update-hook nil)

(add-to-list 'non-printing-types 'fake)
\end{lisp}

\begin{notate}{On `scholia-display-extras-hook'} \label{scholia-display-extras-hook}
This hook is run at the end of `scholia-display-extras'
(Note \ref{scholia-display-extras}) and allows for easy
tweaking of the scholia display (for example, displaying
special sorts of scholia for articles with certain types).

Some basic additions to this hook appear in Section
\ref{linear-browsing}.
\end{notate}

\begin{notate}{On `scholia-display-extras'} \label{scholia-display-extras}
This is used to add some extra stuff to the Scholia Display for
purposes of navigation and establishing context.  It is called by
`display-article' (Note \ref{display-article}) and
`display-scholia-about-current-buffer' (Note
\ref{display-scholia-about-current-buffer}).

It might be nice to have this in the main article window, but that
could also be confusing, especialy if the main article is associated
with some buffer; so I'm taking the route of caution here.  Also note
that according to the principle of order \ref{principle-of-order} this
stuff should probably appear much later on in the document, perhaps in
Section \ref{linear-browsing}.

It might be more appropriate to have some of these extra features
display in or above the main article buffer; the info system has a
nice un-editable bar for display of various navigational data.
\end{notate}

\begin{lisp}
(defun scholia-display-extras ()
  ;; this setting is useful for presentations.
  (goto-char (point-min))
  ;; Be careful that this is ignored when the article is saved or
  ;; otherwise processed internally.
  (insert
   "Title: "
   (propertize (format "%s" name-of-current-article) 'face 'italic)
   "\n\n")
  (goto-char (point-max))
  (run-hooks 'scholia-display-extras-hook)
  (goto-char (point-min)))
\end{lisp}

\begin{notate}{More than one parent} \label{more-than-one-parent}
We'll want to allow \emph{more than one} parent eventually; compare
what happens when following a scholium/reference in a region that has
more than one of these things written about it.  For the purposes of
markup, ``parent'' may become a variable (so add `(reference-to-parent
foo)' for the foo parent) and maybe, if the various reference scholium
objects created here are actually helpful, make them include lists of
references?  Multi-references haven't been done anywhere in this code
so far, but they seem like a reasonable idea.  See also Note
\ref{forward-in-parent}, Note \ref{backward-in-parent}.
\end{notate}

\begin{notate}{Text for display purposes only} \label{display-purposes-only}
I think the setup will work so that individual sections that bear
identification markup will all have their changes propagated properly,
i.e., independently.  However, I think we may have to do a little more
work to make sure that the name of the article (currently shown in
italics whenever an article is displayed, though it would probably be
nicer to emulate the style used by {\tt info}) and any other
after-the-fact textual additions are ignored whenever the article is
saved.
\end{notate}


\subsubsection{Display styles}

\begin{notate}{On `display-style'} \label{display-style}
As discussed in Section \ref{stylistic-considerations}, there are a
number of different styles we'd like to offer users to choose between
when they go to display an article.  For now, the relevant settings
for the display style variable are `plain' (the default, displaying
all scholia) and `contextual' (which causes only those scholia
associated with the region being displayed to appear; see Note
\ref{contextual-display}).

There are a few problems here: if we go with the contextual display,
what about scholia that apply to the article as a whole?  That's a
little tricky.  I guess for now we just leave them out?

Set this variable with `set-display-style', not manually.
\end{notate}

\begin{lisp}
(defvar display-style 'plain)
\end{lisp}

\begin{notate}{On `window-displayed-substring'}
This variable will hold the string that is being shown
through a given window at a given point in time.
\end{notate}

\begin{lisp}
(defvar window-displayed-substring nil)
\end{lisp}

\begin{notate}{On `set-display-style'} \label{set-display-style}
Use this function to control the setting of `display-style' (Note
\ref{display-style}).  Relevant arguments are `plain', or
`contextual'.

(Actually, anything but `contextual' will set `display-style' to
its default setting.  Eventually, we might want to be able to
have contextual display together with some other non-default features,
at which point we'll have to adjust this function appropriately.)
\end{notate}

\begin{lisp}
(defun set-display-style (style)
  (cond ((eq style 'contextual)
         ;; we may need a "double hook" here, so we get the local
         ;; hooks set up in the correct window
         (add-hook 'display-article-hook
                   'initiate-contextual-updating)
         (setq display-style 'contextual))
        (t
         (remove-hook 'display-article-hook
                      'initiate-contextual-updating)
         (setq display-style 'plain))))
\end{lisp}

\begin{notate}{On `initiate-contextual-updating'}
This function gets the rendering target buffer set up
to do live-updating of scholia.

For now, this function doesn't do anything for labels -- since
for now, labels typically don't have scholia on their \inp{text}.
However, this can be undone later, if necessary.

In addition, we don't yet have any code for getting rid of contextual
updating in an individual buffer OTF.
\begin{idea}
(remove-hook 'window-scroll-functions
             'set-window-displayed-substring t)
\end{idea}
\end{notate}

\begin{lisp}
(defun initiate-contextual-updating ()
  (unless (typedata-includes (scholium-type article) 'label)
    (save-excursion (set-buffer rendering-target-buffer)
                    (add-hook 'window-scroll-functions 
                              'maybe-update-scholia-display nil t))))
\end{lisp}

\begin{notate}{On `maybe-update-scholia-display'} \label{maybe-update-scholia-display}
In order for this to work optimally, we'd might to keep track of
a list of the current scholia that are being displayed.  But
for now, I suppose (contrary to this function's name) we can just
redisplay all the scholia we encounter every time.
\end{notate}

\begin{lisp}
(defun maybe-update-scholia-display ()
  (set-window-displayed-substring)
  ;; this is just a ridiculous thing to run -- for testing purposes
  ;; only.  Eventually, we'll actually want to analyse the text, 
  ;; figure out which scholia are relevant, and update the scholia
  ;; display.
  (save-excursion (set-buffer (get-buffer-create "*scratch*"))
                  (erase-buffer)
                  (insert window-displayed-substring)))
\end{lisp}

\begin{notate}{On `set-window-displayed-substring'}
This function will be an element of the `window-scroll-functions'
hook when the display mode is set to `contextual'.
\end{notate}

\begin{lisp}
(defun set-window-displayed-substring
  (setq window-displayed-substring
        (buffer-substring (window-start)
                          (window-end))))
\end{lisp}

\subsubsection{Displaying articles}

\begin{notate}{On `display-article-hook'}
We will later do some ``interesting'' things with this function, so we
add a hook.  For example, one use for this hook is to maintain a
history of articles that have been displayed; see Section
\ref{temporal}.
\end{notate}

\begin{lisp}
(defvar display-article-hook nil)
\end{lisp}

\begin{notate}{On `raw-scholia-selector'} 
This function is here both to select raw scholia for display when
`display-article' runs, and to provide for a choice between different
ways of selecting raw scholia.  It returns a list of scholia.

This function is set up to run within the scope of `display-article' 
(Note \ref{display-article}).

We select slightly different scholia when the article that is being
displayed is a label; in particular, we don't want to display scholia
that indicate the current article as a `parent', since these scholia
will already be listed in the label's \inp{text} itself.

If we're trying to display scholia contextually, then this function
should probably be doctored with some, to make it so that the initial
set of scholia that are displayed are contextually appropriate, i.e.,
are just those associated with the on-screen portion of the buffer
that is being displayed.
\end{notate}

\begin{lisp}
(defun raw-scholia-selector (&optional what-is-displayed)
  (cond ((eq what-is-displayed 'label)
         (remove-if                
          (lambda (scholium)
            (member-if (lambda (link)
                         (member 'parent (cdr link)))
                       (scholium-about scholium)))
          (mapcar (lambda (backlink)
                    (get-article (car backlink)))
                  (get-backlinks name-of-current-article))))
        (t (mapcar (lambda (backlink)
                     (get-article (car backlink)))
                   (get-backlinks name-of-current-article)))))

;; (eq display-style 'contextual)
;;          (save-excursion (set-buffer (get-buffer-create "*scratch*"))
;;                          (erase-buffer)
;;                          (insert window-displayed-substring))
;;          nil
\end{lisp}

\begin{notate}{On `display-article'} \label{display-article}
Display article found via \inp{path} (which can just be the name of an
article in the main article tabel), if said article exists.  If the
article lives in a buffer, that buffer will be where the article is
displayed; otherwise the article is displayed in the ``Main Article
Display'' buffer.  Note that this function can also be used to display
labels (it calls `display-label'; but see Note
\ref{display-scholia-about-labels}, since some other approach might be
valuable sometimes).

I don't want to have all of the children appear as scholia when a
label is browsed, at least not by default; but it is kind of neat to
know that they can be made to appear; if we don't do the `remove-if'
then the scholia display will be the OTF-assembled compilation
mentioned in Note \ref{bits-and-pieces}.

It makes more sense to always run `display-article-hook' after the
`cond', and to put things that are conditional upon being in this
specific branch into `scholia-display-post-update-hook' or, barring
that, some additional branch-specific hook.  This of course means that
the parent will be added to the history list if that is where we
browse from; this is intended.

If it turns out to be needed here, we could reuse the trick of running
a hook and then making a test before the `cond', which we're familiar
with from e.g. `sch-plain-text' (Note \ref{sch-plain-text}).

It would be good to make the buffer disposition (left? right?)
is consistent when we use this function to display labels;  I
think we have it sorted out properly for the display of 
``normal'' articles.

We may want to treat namespaces and labels together in the same
`cond' branch (currently namespaces aren't handled specially by
this function). 

Since we are now reading in paths, these paths have to be parsed.
This is the job of `generalized-get-article' (see Note
\ref{generalized-get-article}).
\end{notate}

\begin{lisp}
(defun display-article (path)
  (interactive (list (read-article-path)))
  (let* ((article (generalized-get-article path))
         (name (scholium-name article)))
    (if (not article)
        (error "No article by that name found")
      (setq name-of-current-article name)
      (cond
       ((typedata-includes (scholium-type article) 'label)
        (display-label name)
        (setq rendering-target-buffer "*Generic List*")
        (switch-to-buffer (get-buffer "*Generic List*"))
        (setq buffer-read-only nil)
        (switch-to-buffer (get-buffer-create "Scholia Display") t)
        (erase-buffer)
        (let ((raw-scholia (raw-scholia-selector 'label)))
          (mark-things-up))
        (scholia-display-extras)
        (switch-to-buffer (get-buffer "*Generic List*"))
        (setq buffer-read-only t)
        (pop-to-buffer "Scholia Display")
        (other-window -1))
       (t
        (if (article-buffered article)
            (setq rendering-target-buffer (scholium-text article))
          (setq rendering-target-buffer nil))
        (run-hooks 'scholia-display-pre-update-hook)
        (scholia-overwhelm-display (sch-plain-text article))
        (let ((raw-scholia (raw-scholia-selector)))
          (mark-things-up))
        (scholia-display-extras)
        (pop-to-buffer (get-rendering-target-buffer))
        (run-hooks 'scholia-display-post-update-hook)))
      (run-hooks 'display-article-hook))))
\end{lisp}

\begin{notate}{Argument of `display-article' is typically a string}
Note that for typical interactive use, a string is expected, which
means a string is produced, which means that when `get-article' is
called in the next line, bad things will happen for articles whose
names are not strings -- unless we do something about it.
\end{notate}

\begin{notate}{Would have liked to associate a type with paths} \label{associate-a-type-with-paths}
It would be nice to use a special type of object to distinguish
between paths and names.  (I.e., if `read-article-path' returns an
object of type path when it actually builds a path and
`generalized-get-article' looks for this type before deciding what to
do, we could be OK just passing in an article name to the generalized
function.)  But I'm not sure how to assert that a given form has a
given type (or even whether this is possible).  Maybe by working with
structs one could do it, but I don't understand structs (their
documentation seems to be lacking).

Note, the idea of associating a type with a form seems very
scholiumific.  But I've forgotten whether Lisp has anything for doing
this.
\end{notate}

\begin{notate}{On `redisplay-article'}
This accomplishes a simple task.

It would be nice if we could get the point restored to
its original position after this thing runs.

Also, it would be good to have the function run automatically
after scholia have been added about the document.
\end{notate}

\begin{lisp}
(defun redisplay-article ()
  (interactive)
  (display-article name-of-current-article))
\end{lisp}

\subsubsection{Displaying scholia about a given buffer}

\begin{notate}{On `display-scholia-about-current-buffer'} \label{display-scholia-about-current-buffer}
This function is similar to `display-article' (\ref{display-article}),
but it works directly on the buffer level.  (Stylistically this
function should probably just be a thin wrapper, but we're running
with it for now.)

This function should perhaps check to see whether the
current buffer has been \emph{edited} since the last time
this function (or similar, through other means) was executed.
If the buffer has been edited, the user should probably
be prompted, and asked whether to reparse (Section \ref{parsing})
before redisplaying.
\end{notate}

\begin{lisp}
(defun display-scholia-about-current-buffer ()
  (interactive)
  (let ((article (get-article (buffer-name (current-buffer)))))
    (if article
        (progn
          (setq rendering-target-buffer (current-buffer))
          (run-hooks 'scholia-display-pre-update-hook)
          (scholia-overwhelm-display (sch-plain-text article))
          (let ((raw-scholia
                 (mapcar (lambda (backlink)
                          (get-article (car backlink)))
                         (get-backlinks name-of-current-article))))
            (mark-things-up))
          (scholia-display-extras)
          (pop-to-buffer (get-rendering-target-buffer))
          (run-hooks 'scholia-display-post-update-hook)
          ;; weird!
          (run-hooks 'display-article-hook))
      (call-if-user-adds-current-buffer-to-article-list
       'display-scholia-about-current-buffer))))
\end{lisp}

\begin{notate}{Running `display-article-hook' from `display-scholia-about-current-buffer'?}
It seems a bit little weird to have to run this hook here.
\end{notate}

\begin{notate}{Automatically display scholia about any found buffer}
It might be handy to integrate the scholium system into
day-to-day Emacs operation by adding
`display-scholia-about-current-buffer' to the `find-file-hook'.
\end{notate}

\begin{notate}{Displaying a generic buffer together with scholia}
You should be able to simply call
`display-scholia-about-current-buffer' after loading the buffer
(\emph{untested}).  To turn it on all the time, add it to the
appropriate hook.  Note that this won't add any new scholia to the
system.  (To add new scholia at \emph{file} load time, we'll need some
additional code.)
\end{notate}

\subsubsection{Displaying labels} \label{displaying-labels}

\begin{notate}{On `display-label-hook'}
This provides a way to customize the behavior of
`display-label'.
\end{notate}

\begin{lisp}
(defvar display-label-hook nil)
\end{lisp}

\begin{notate}{On `currently-displayed-label'}
This gives us a handle on the most recently displayed
label.  This facilitates only the simplest 
improvement to reverting behavior.
\end{notate}

\begin{lisp}
(defvar currently-displayed-label nil)
\end{lisp}

\begin{notate}{On `display-label'} \label{display-label}
The function `display-label' uses the catalog browsing feature of
Section \ref{catalog}.  If we want to do other interesting rendering
things with articles that have special types, we can follow the usage
of `display-label' in `display-article'.  It might be good for this
function to run its own hook, e.g., for maintaining a special history
(see \ref{history-for-listing}).
\end{notate}

\begin{lisp}
(defun display-label (name)
  (interactive 
   (list
    (let* ((completion-ignore-case t)
           (label-names
            (scholium-text (get-article 'label)))
           (label-strings (mapcar (lambda (name) 
                                    (format "%s" name))
                                  label-names))
           (string-found (completing-read
                          "Label: "
                          label-strings))
           (place (- (length label-strings)
                     (length (member string-found label-strings)))))
      (nth place label-names))))
  (article-menu-listing (label-to-propertized-list name))
  (run-hooks 'display-label-hook))

(add-hook 'display-label-hook (lambda ()
                                (setq currently-displayed-label name)))

(defalias 'list-label 'display-label)
\end{lisp}

\begin{notate}{On `display-intersection-of-labels'} \label{display-intersection-of-labels}
For displaying everything bearing every one of the
input \inp{labels}.
\end{notate}

\begin{lisp}
(defun display-intersection-of-labels (&rest labels)
  (let ((intersection (label-to-list (car labels)))
        (ctn (cdr labels)))
    (while ctn
      (setq intersection (intersection intersection
                                       (label-to-list (car ctn))
                                       :test 'equal))
      (setq ctn (cdr ctn)))
    (article-menu-listing (turn-list-into-propertized-list
                           intersection))))
\end{lisp}

\begin{notate}{On `display-difference-of-labels'} \label{display-difference-of-labels}
For purposes of simplicity, this is set up to work with two labels
only, for the time being.
\end{notate}

\begin{lisp}
(defun display-difference-of-labels (label-A label-B)
  (article-menu-listing (turn-list-into-propertized-list
                         (set-difference (label-to-list label-A)
                                         (label-to-list label-B)))))
\end{lisp}

\begin{notate}{Label derivatives} \label{label-derivatives}
It might be the case that someone
would want to add a new label to the elements of an intersection
or difference (as would be found by internal routines of
`display-intersection-of-labels' and `display-difference-of-labels').
\end{notate}

\begin{notate}{Display type in another column}
Probably it would be good to have another accessor that would
show the article's type in a second column.  This could be
especially useful in a context in which a section can contain
both notes and subsections (just for example).
\end{notate}

\begin{notate}{Display scholia about labels too, at least sometimes} \label{display-scholia-about-labels}
It would probably be good to be able to be able to compose and
display scholia about labels, since they are, after all,
also articles.  Probably this should be optional, since there are
times when we just want to see the list and not think of it as
an article.  And in addition, if we do plan to display scholia
about labels only sometimes, we'll have to decide what to do with
the contents of the scholia display buffer at times when we just
want to display the list and not its scholia.  One trick might be to
make the listing prefer to appear in the window wherein the scholia
are displayed in such a case, if it exists; this way both windows
wouldn't confusingly appear at the same time.  (Compare Note
\ref{redisplay-after-running-make-scholium}.)
\end{notate}

\subsubsection{Finding marked regions} \label{finding-marked-regions}

\begin{notate}{On `find-marked-regions'}
This function is used in the subsequent section (Section
\ref{overlays-on-off}) to locate the parts of the buffer
that have scholia written about them.  Note that
this scheme might have been outmoded by stuff in Section
\ref{parsing}.
\end{notate}

\begin{lisp}
(defun find-marked-regions ()
  (let (names-and-positions
        (next-change-point (point-min)))
    (while next-change-point
      (let ((next-region (find-next-marked-region)))
        (when next-region
          (setq names-and-positions
                (cons next-region
                      names-and-positions)))))
    names-and-positions))
\end{lisp}

\begin{notate}{On `find-next-marked-region'}
This function is within the scope of `find-marked-regions'.  Should be
able to find all the regions associated with any scholium.  Right now,
this function is working in a simplified universe in which scholia and
regions are mapped to each other in 1-1 way!
\end{notate}

\begin{lisp}
(defun find-next-marked-region ()
  (let* ((beg (next-single-property-change next-change-point
                                           'scholia))
         (end (when beg
                (next-single-property-change beg
                                             'scholia))))
    (setq next-change-point end)
    (when end
      (list
       (get-text-property beg 'scholia)
       (list (cons beg end))))))
\end{lisp}

\subsubsection{Turning overlays off and on} \label{overlays-on-off}

\begin{notate}{Overlays exercises}
As an exercise with text properties and overlays, here is some code
for turning the overlays on and off in the Scholium Display and main
article buffers.
\end{notate}

\begin{notate}{Turning overlays off}
It is easy enough to turn overlays off; this is accomplished for the
main article buffer and the Scholia Display buffer by
`sch-turn-main-article-overlays-off' and
`sch-turn-scholia-overlays-off', respectively.
\end{notate}

\begin{lisp}
(defun sch-turn-main-article-overlays-off ()
  (interactive)
  (mapcar #'delete-overlay main-article-overlays)
  (setq main-article-overlays nil))

(defun sch-turn-scholia-overlays-off ()
  (interactive)
  (mapcar #'delete-overlay scholia-overlays)
  (setq scholia-overlays nil))
\end{lisp}

\begin{notate}{Turning overlays on}
Turning overlays on is a bit trickier.  In order to be able to turn
scholia on, we need to be able to find all the regions that have
scholia attached to them.  This is accomplished (for the main article
buffer only, I think) by `find-marked-regions'.  
\end{notate}

\begin{lisp}
(defun sch-turn-main-article-overlays-on ()
  (interactive)
  ;; to save from potential overlap weirdness
  (sch-turn-main-article-overlays-off)
  (save-excursion
    (let ((names-and-positions (find-marked-regions)))
        (dolist (info names-and-positions)
    (let* ((name (car info))
           (marked-regions (cadr info))
           (scholium (get-article name))
           (beg (point)))
      (save-excursion
        (set-buffer-to-rendering-target-buffer)
        (dolist (reg marked-regions)
          ;; add "transient" overlay
          (when use-crazy-font-lock
            (setq main-article-overlays
                  (cons
                   (make-overlay (car reg)
                                 (cdr reg)
                                 (get-rendering-target-buffer)
                                 t)
                   main-article-overlays))
            ;; using `scholium-face' here is a bit weird
            (overlay-put (car main-article-overlays)
                         'face (scholium-face))))))))))

(defun sch-turn-scholia-overlays-on ()
  (interactive)
  )
\end{lisp}

\begin{notate}{Using these functions as a subroutine for initial markup?}
It doesn't make a whole lot of sense to use these functions for
initial markup, because they involve a bit of search that would be
unneeded at that stage.  However, it would be good to take another
look at `mark-things-up' and see if we can get any further insights
into how to write this function from there.  (Presumably this function
was initially based on that one already...)
\end{notate}

\section{Browsing} \label{browsing}

\subsection{Scholia browsing} \label{browse-scholia}

\begin{notate}{Introduction to scholia browsing}
The point of this section is to associate scholia with the regions of
the article that they apply to, and to enable the user to quickly find
and navigate between the marked-up regions in the displayed version of
this article and their associated scholia.
\end{notate}

\begin{notate}{Moving without reparsing} \label{moving-with-reparsing}
The functions for scholia browsing are likely to move you to the
``wrong place'' unless the buffer has been parsed and the scholia
locations are up-to-date (i.e., it will move you to the specified
region, but this will probably be the wrong region).
\end{notate}

\begin{notate}{Problem with the design of the `move-to...-scholium' functions}
These probably shouldn't be using overlays as the source of their
 information about where scholia reside, since overlays are not
 necessarily going to be there!  

(I guess this means ``if the content has been cut and pasted.''  I'm
 guessing that there are cases where we would like to use this
 function under conditions in which cutting and pasting may have taken
 place.  I'm not sure exactly what those cases would be, since I'm not
 sure exactly when this function is called.  Seems it is, so far, only
 called in the case of scrolling the main article display to a certain
 region, namely a region with an attached scholium.  It might be
 reasonable just to use overlays in that case (?).  This is a somewhat
 curious point, and since it actually deals with something sort of
 fundamental in Emacs, maybe it would be best to give it more thought
 \emph{later}, especially since this function is probably useful right
 \emph{now}.)

They can be turned off altogether (Section \ref{overlays-on-off}) or
 one by one (Note \ref{overlay-styles}).

(That is a good point.  Audience, what do you think?)

Anyway, the logic of the function is probably going to be similar
no matter what guts we use.

Note that there is a funny case of two references that begin
at the same point (and, possibly, end at the same point as well).
I think this case isn't handled well by the function the way it
is written.

This is a general matter that goes beyond these functions -- and
 should probably be noted as such somewhere.
\end{notate}

\begin{notate}{On `move-to-next-region-with-scholium' and `move-to-previous-region-with-scholium'} \label{move-to-next-or-previous-region-with-scholium}
This moves the point to the beginning of the next region that has a
scholium about it (if there is one).

(I'm noticing a bug when the function is used interactively with the
binding selected in Section \ref{bindings} and the cursor is
positioned on a right paren; calling the function with M-x in this
case doesn't result in the same problem.)

Note these two functions are not quite symmetrical, because we
want the cursor to end up at the beginning of the marked
region.  (I think there will be a problem if we try to go to the
beginning of a scholium that is attached at the beginning of
the article, but that isn't such a big deal.)
\end{notate}

\begin{lisp}
(defun move-to-next-region-with-scholium ()
  (interactive)
  (let ((change (next-overlay-change (point))))
    (if (overlays-at change)
        (progn (goto-char change)
               (list change (next-overlay-change change)))
      (if (overlays-at (next-overlay-change change))
          (progn (goto-char (next-overlay-change change))
                 (list (next-overlay-change change)
                       (next-overlay-change (next-overlay-change 
                                             change))))
        (message "No subsequent regions with scholia about them.")
        nil))))

(defun move-to-previous-region-with-scholium ()
  (interactive)
  (let ((change (previous-overlay-change (point))))
    (if (overlays-at change)
        (progn (goto-char change)
               (list change (previous-overlay-change change)))
      (if (overlays-at (previous-overlay-change change))
          (progn (goto-char (previous-overlay-change change))
                 (list (previous-overlay-change change)
                       (previous-overlay-change (previous-overlay-change 
                                             change))))
        (message "No previous regions with scholia about them.")
        nil))))

(defun scroll-article-display-to-next-region-for-current-scholium ()
  (interactive)
  (save-excursion
    (set-buffer buffer-associated-with-current-article)
    (let (found
          (curpoint (point)))
      (while (and (not found)
                  (not (eobp))
                  (move-to-next-region-with-scholium))
        (mapc (lambda (overlay)
                (when (equal (overlay-get overlay 'scholia)
                             (name-of-current-scholium))
                  (setq found t)))
              (overlays-at (point)))
        (if found
            (recenter)))
      (when (not found)
        (goto-char curpoint)
        (message "Scholium not about further regions in buffer.")))))

(defun scroll-article-display-to-previous-region-for-current-scholium ()
  (interactive)
  (save-excursion
    (set-buffer buffer-associated-with-current-article)
    (let (found
          (curpoint (point)))
      (while (and (not found)
                  (not (bobp))
                  (move-to-previous-region-with-scholium))
        (mapc (lambda (overlay)
                (when (equal (overlay-get overlay 'scholia)
                             (name-of-current-scholium))
                  (setq found t)))
              (overlays-at (point)))
        (if found
            (recenter)))
      (when (not found)
        (goto-char curpoint)
        (message "Scholium not about previous regions in buffer.")))))
\end{lisp}

\begin{notate}{Cycling through regions}
It wouldn't be too hard to make a variant of
`scroll-article-display-to-previous-region-for-current-scholium' that
cycles through the regions that the current scholium is about.
\end{notate}

\begin{notate}{On `move-to-first-region-for-scholium'} \label{move-to-first-region-for-scholium}
Move you to the beginning of the region marked up by the scholium
named \inp{name}.  Should this be made interactive?)  At present, it
is only called by `move-to-first-region-for-current-scholium'.
\end{notate}

\begin{lisp}
(defun move-to-first-region-for-scholium (name)
  (pop-to-buffer (get-buffer rendering-target-buffer))
  (let ((beg (point-max))
        (about (scholium-about (get-article name))))
    (dolist (link about)
      ;; this should be revised in light of the
      ;; fact that a link can be multiply typed
      (when (and (typedata-includes-passage (link-type elt))
                 (equal (linked-to-article link)
                        name-of-current-article)
                 (< (link-beginning link) beg))
        (setq beg (link-beginning link))))
    (unless (equal beg (point-max))
      (goto-char beg))))
\end{lisp}

\begin{notate}{On `move-to-first-region-for-current-scholium'}
This uses the function `move-to-first-region-for-scholium' from
section \ref{browse-scholia}; the thought behind including the
function here is that it establishes a relationship between
the Scholia Display buffer and the main article buffer (however
it could probably go in section \ref{browse-scholia} equally well).

This should probably be complemented by a function
`move-to-last-region-for-current-scholium'.

Also, it should probably have some intelligent message
(not to say ``error message'') if the scholium applies to
the article as a whole.

(Gives some error, complaining about `elt' being void.
Can this run in the Scholia Display buffer as well as the
main article buffer?)
\end{notate}

\begin{lisp}
(defun move-to-first-region-for-current-scholium ()
  (interactive)
  (let ((current (name-of-current-scholium)))
    (move-to-first-region-for-scholium current)))
\end{lisp}

\subsection{Local browsing} \label{local}

\begin{notate}{Introduction to local browsing}
This section will describe purely local navigation commands that
require little in the way of additional data structures.  The main
idea for the functions in this section is to make it so that focus can
move from the current article to adjacent articles (either articles
that are about the current article or articles that the current
article is about), or more generally, to articles within some
neighborhood of the current article (articles that we can reach by
stepping from adjacent article to its adjacent articles, etc.).  We
re-use the listing mechanism from the previous section as needed.
\end{notate}

\begin{notate}{On `read-scholia-property-at-point'}
Suppose we simply want to make the current scholium into the new
current article.  That's what the next function is for.
\end{notate}

\begin{lisp}
(defun read-scholia-property-at-point ()
  (get-text-property (point) 'scholia))
\end{lisp}

\begin{notate}{On `scholia-named-at-point'}
I think that this should strip out the ``mask'' tags
from the link-ids, but leave the name parts.  Since it
is only used by interactive functions, this seems fine,
and appropriate.
\end{notate}

\begin{lisp}
(defun scholia-named-at-point ()
  (mapcar (lambda (id)
            (if (eq (car id) 'mask)
                (car (second id))
              (car id)))
          (read-scholia-property-at-point)))
\end{lisp}

\begin{notate}{On `follow-scholium'} \label{follow-scholium}
This causes the current scholium to become the current article.

Eventually we'll want to be able to run this command with a
mouse-click.
\end{notate}

\begin{lisp}
(defun follow-scholium ()
  (interactive)
  (let ((current (name-of-current-scholium)))
    (when current
      (display-article current))))
\end{lisp}

\begin{notate}{On `follow-reference'} \label{follow-reference}
This reads the `scholia' property at point and either follows the
reference at point (if there is only one) or allows the user to choose
between references (if there are several).

We might want to provide an additional function for following links
\emph{in general}; basically the strategy for that is, just don't do
the `remove-if' filtering.
\end{notate}

\begin{lisp}
(defun follow-reference ()
  (interactive)
  (let ((references
         (remove-if (lambda (name)
                      (not (typedata-includes
                            (scholium-type (get-article name))
                            'reference)))
                    (scholia-named-at-point))))
    (cond
     ((equal (length references) 1)
      (let* ((ref (get-article (car references)))
             (to-article (reference-to-article ref)))
        (if (equal to-article name-of-current-article)
            (display-article (reference-from-article ref))
          (display-article to-article)))
      ;; maybe `display-article' should be returning
      ;; some non-`nil' value so that we don't have to do this.
      t)
     (references
      ;; this sort of disambiguation is really only needed if the
      ;; references have different targets.  Two distinct references
      ;; to the same thing overlaying each other could be treated as
      ;; one for simple following purposes.
      (list-articles references))
     (t
      (message "No reference at point.")
      nil))))

(defun follow-reference-or-scholium ()
  (interactive)
  (unless (follow-reference)
    (follow-scholium)))

(defun display-an-article-that-current-article-is-about ()
  (interactive)
  (let ((abouts (scholium-about
                 (get-article name-of-current-article))))
    (cond
     ((equal (length abouts) 1)
      (display-article (car abouts)))
     (abouts
      (list-articles abouts))
     (t
      (message "Article isn't about any other articles.")))))
\end{lisp}

\begin{notate}{On `current-scholium-is-about'}
This function is similar to the previous one, but it applies to
scholia.  Since one presumably knows that the current scholium is
about the current article, this is most useful when a scholium is
about several different articles, as it allows the user to move
``down'' to any of them.
\end{notate}

\begin{lisp}
(defun current-scholium-is-about ()
  (scholium-about (get-article (name-of-current-scholium))))

(defun display-an-article-that-current-scholium-is-about ()
  (interactive)
  (let ((abouts (current-scholium-is-about)))
    (cond
     ((equal (length abouts) 1)
      (display-article (car abouts))
      (message "Note: scholium is only about current article."))
     (abouts
      (list-articles abouts))
     (t
      (message "Article isn't about any other articles.")))))
\end{lisp}

\subsection{Catalog browsing} \label{catalog}

\begin{notate}{Content-free browsing}
A popular style of browsing ignores the content of the objects
themselves and looks only at their metadata.  Examples include library
catalogs or lists of search engine results.  Since you aren't actually
browsing the articles directly, this is ``browsing a catalog'' more
than it is ``browsing the library''.

Using this technique, one should be able to selectively choose
subcollections to look at, e.g. all the works by a given author or
what have you.  In this section, we provide code for browsing by
article name.  The code is an interface to a generic browsing/listing
command based on Emacs's built-in `list-buffers' command.  This setup
is designed with extensibility in mind, however it may be useful to
rewrite the selection mechanism (or add an alternate version) in order
to make each line in the current display into a bibliography-style or
index-card style reference.  This would make the information fit on
the screen better (and such modifications should be easy to make
later).

See Note \ref{standard-article-menu-accessors} for a description of
the data that are displayed along with names in the listing.  We may
eventually want to be able to switch between several alternative
data-listings, e.g., we might want name, type, and namespace; or name,
article-about list, about-article list.  With if complicated lists
like this are used, we may want to have more than one line associated
with each name.

We currently provide only very basic functions for working with these
listings.
\end{notate}

\begin{notate}{Catalog browsing in the scholium system}
This code provides a generic way to perform actions on rich objects
from a list.  It is assumed that each object has several attributes
that a user may be interested in.  ``Name'', ``size'' and ``mode'' are
familiar examples from `list-buffers'; here there can be an
arbitrary number of arbitrary attributes.  This makes this
mechanism useful in some cases where `completing-read' would not
provide the user with enough information.

The way attributes are obtained from objects is left up to the
user, as are the actual actions that can be performed on the
objects.

Generality comes at an obvious, unavoidable, cost, namely any given
deployment of this interface requires some extra set up -- examples
are provided.
\end{notate}

\begin{notate}{Predicate support for listing} \label{predicate-support-for-listing}
Just for example, one thing people might want to do would be to list
the articles in the vicinity of the article being browsed or in a
certain section from a paper, and so on.  The function will work with
any set that we can select; what we need are a bunch of functions for
doing the selection.
\end{notate}

\begin{notate}{History support for listing} \label{history-for-listing}
It would be good to maintain a special history list for this display,
that way if you browse a succession of nested catalogs, you can go
backwards and forwards to choose the best catalog.  (Catalogs are
similar to directories from dired, gopher, etc.)  See Section
\ref{temporal} for a discussion of history support in general (start
with Note \ref{thematic-histories}).
\end{notate}

\begin{notate}{Purpose of generality}
Different menus may be useful for different purposes.  A collection of
article \inp{name}s may frequently be insufficient data from which to
choose an article.  Different data may be useful for different times
and purposes, so we've made the system extensible.  It may be useful
to change the overall \emph{style} of the display as well (e.g. to
select from a list of multi-line records rather than a list of lines);
modifications at this level are on hold until such time as they are
needed.
\end{notate}

\begin{notate}{On `make-generic-menu-mode'}
This takes the name of the menu (as a space-separated string) together
with a list of bindings to be used in that particular menu mode.

This should provide a docstring for the mode it creates.
\end{notate}

\begin{lisp}
(defmacro make-generic-menu-mode
  (mode bindings)
  (let* ((modedash (downcase (replace-regexp-in-string " " "-" mode)))
         (modesymbol (intern (concat modedash "-mode")))
         (mapname (intern (concat modedash "-map"))))
    `(progn
       (defvar ,mapname)
       (setq ,mapname (make-keymap))
       (suppress-keymap ,mapname t)
       (dolist (binding ,bindings)
         (define-key ,mapname (car binding) (cdr binding)))
       (defun ,modesymbol ()
         (kill-all-local-variables)
         (use-local-map ,mapname)
         (setq major-mode (quote ,modesymbol))
         (setq mode-name ,mode)
         (setq truncate-lines t)
         (setq buffer-read-only t)))))
\end{lisp}

\begin{notate}{On `generic-menu-noselect'} \label{generic-menu-noselect}
The basic idea of this is that we have some \inp{objects} and some
functions to map across the objects to extract the information from
the objects (\inp{accessors}).  The functions must be set up to
produce strings as their output.  The functions correspond to columns
in the display; individual objects correspond to rows.  This is the
same idea no matter what the source of the objects is.  (In
particular, it might be a good idea for a later version of this
function to accept either a hash table or a list as the source of the
objects; see Note \ref{browsing-subcollections}.)  Note that columns
of the display are assumed to be as wide as their widest item!
\end{notate}

\begin{lisp}
(defun generic-menu-noselect (objects accessors)
  (let (cols)
    (dolist (get-this accessors)
      ;; if we built this front-to-back rather than back-to-front,
      ;; that would be better
      (setq cols (cons
                  (cons (car get-this)
                        (mapcar (cdr get-this) objects))
                  cols)))
    ;; find the width of the columns.
    (let ((lens (mapcar (lambda (col)
                          (let ((len 0))
                            (dolist (str col)
                              (let ((candidate (length str)))
                                (when (> candidate len)
                                  (setq len candidate))))
                            len))
                        cols)))
      (with-current-buffer (get-buffer-create "*Generic List*")
        (setq buffer-read-only nil)
        (erase-buffer)
        (while cols
          (goto-char (point-min))
          (goto-char (line-end-position))
          (dolist (str (car cols))
            (insert str " ")
            ;; fill with spaces to make up for lost space
            (insert-char 32 (- (car lens) (length str)))
            (unless (equal (forward-line) 0)
              (newline))
            (goto-char (line-end-position)))
          (setq cols (cdr cols))
          (setq lens (cdr lens)))
        (goto-char (point-min))
        (current-buffer)))))
\end{lisp}

\begin{notate}{Improved sorting of article listing}
Sorting could be done using autocompletion on the name of the column
to sort on, too, which would be kind of nice.  Or we could make a
command to sort on the current column, or reorder columns according to
the values on the current line or just about anything you might like.

But I don't think sorting is going to work at all until we have a
consistent way of identifying the fields to sort; in the case of
strings with spaces in them, `sort-fields' won't work.  Since we
compute the width of each of the columns in `generic-menu-noselect',
if we were to store this info (perhaps as a text property attached to
each column heading), we could adroitly divide the text up to find the
strings we're trying to sort.  See Note \ref{text-properties-in-article-listing}.
\end{notate}

\begin{lisp}
(defun Generic-menu-sort (col)
  (interactive "P")
  (save-excursion
    (sort-fields (or col 1) (progn (goto-line 2)
                                   (point))
                 (point-max))))
\end{lisp}

\begin{notate}{Hierarchies for browsing}
Hierarchies are somewhat similar to menus, e.g., we might
want to use hierarchies to display sub-library relationships.
(This is like KM's ontology browser.)  Of course, graph
browsing would be even more generic (but display of these
things seems a bit far removed from what we can do right now).
\end{notate}

\begin{notate}{Reuse of names from article listing}
It would be nice if names were displayed together with quote marks
making the printed versions of the names appropriate for direct
re-use.
\end{notate}

\begin{notate}{On `standard-article-menu-accessors'} \label{standard-article-menu-accessors}
Note that it would be easy to provide more metadata -- just revise
this variable with additional fields as desired.  See Note
\ref{generic-menu-noselect} for a description of how accessors work.
\end{notate}

\begin{lisp}
(defvar standard-article-menu-accessors
  '(("Name" . identity)
    ("C"    . (lambda (elt) " "))))
\end{lisp}

\begin{notate}{Control panels} \label{control-panels}
A little control panel on the left of the article listing comprised of
various toggle-state elements provides a way to interactively
manipulate the articles individually or \emph{en masse} (see section
\ref{masse}).  (The only feature in the control panel for now is an
indicator showing which article is current; this is indicated by a dot
in the ``C'' column.)
\end{notate}

\begin{notate}{On `article-menu-listing-hook'}
This is here to
customize the behavior of the article menu listing.  Currently it
is used to offset the activities invoked by
`currently-displayed-label-hook'; whereas we want
`currently-displayed-label' to be defined when the listing is
used to display a label, we'd rather it be `nil' when something
other than a label has been displayed, since anything else could
be misleading.
\end{notate}

\begin{lisp}
(defvar article-menu-listing-hook nil)
\end{lisp}

\begin{notate}{On `article-menu-listing'}
The optional input \inp{subset} is a list of article names to pump
into the generic menu; it defaults to the list of ``plain'' articles
as recorded on the corresponding label.  The optional input
\inp{accessors} specifies the functions to use to extract information
from the articles named by \inp{subset}; it is in the format of, and
defaults to, `standard-article-menu-accessors'.  (Note that function
plays a similar role to `Buffer-menu-revert' from buff-menu.el.)
\end{notate}

\begin{lisp}
(defun article-menu-listing (&optional subset accessors)
  (interactive)
  (pop-to-buffer
   (generic-menu-noselect
    ;; maybe this should always handle propertizing itself?
    (or subset
        (label-to-propertized-list 'plain))
    (or accessors
        standard-article-menu-accessors)))
  ;; note, this runs every time, even if the current article
  ;; isn't on the list.
  (article-menu-point-out-current-article)
  (article-menu-mode)
  (run-hooks 'article-menu-listing-hook))

(add-hook 'article-menu-listing-hook (lambda ()
                                       (setq currently-displayed-label
                                             nil)))
\end{lisp}

\begin{notate}{On `turn-article-table-into-list'}
I hate to actually use this function the way it is used in
`display-article'... probably we should just select from the
\emph{plain} articles there, and write a separate function to display
other articles.  In short, there really shouldn't be any need to use
this function, except maybe for debugging purposes or \emph{explicit}
listing of all the articles (whenever that really needs to be done).
\end{notate}

\begin{lisp}
(defun turn-article-table-into-list ()
  (let ((names (list t)))
    (maphash (lambda (name value)
               ;; It might be nice to have %S here, but
               ;; I don't know if it would be _useful_
               (nconc names (list (format "%s" name))))
             article-table)
    (cdr names)))
\end{lisp}

\begin{notate}{On `turn-article-table-into-names'}
This is a variant of `turn-article-table-into-list'
that produces the actual names of the articles;
probably it should supercede the other, since we could
get the print names by mapping over the output of this
function, obviously.
\end{notate}

\begin{lisp}
(defun turn-article-table-into-names ()
  (let ((names (list t)))
    (maphash (lambda (name value)
               (nconc names (list name)))
             article-table)
    (cdr names)))
\end{lisp}

\begin{notate}{On `turn-article-table-into-propertized-list'}
This combines the best of both `turn-article-table-into-list' and
`turn-article-table-into-names'.  Compare
`label-to-propertized-list'.
\end{notate}

\begin{lisp}
(defun turn-article-table-into-propertized-list ()
  (let ((names (list t)))
    (maphash (lambda (name value)
               (nconc names
                      (list 
                       (propertize (format "%s" name) 'name name))))
             article-table)
    (cdr names)))
\end{lisp}

\begin{notate}{On `turn-list-into-propertized-list'} \label{turn-list-into-propertized-list}
Turns an arbitrary list into a propertized list.
\end{notate}

\begin{lisp}
(defun turn-list-into-propertized-list (lis)
  (let (names)
    (mapc (lambda (name)
            (setq names (cons
                         (propertize (format "%s" name) 'name name)
                         names)))
          lis)
    names))
\end{lisp}

\begin{notate}{Special-purpose listings} \label{special-listings}
Here are a few functions to list special collections of articles.
The function `article-menu-list-labels' is perhaps particularly
noteworthy; browsing labels seems to be a powerful way of organizing
and retrieving information, see Note \ref{browsing-subcollections}.
\end{notate}

\begin{lisp}
(defun article-menu-list-plain-articles ()
  (interactive)
  (article-menu-listing))

(defun article-menu-list-all-articles ()
  (interactive)
  (article-menu-listing (turn-article-table-into-propertized-list)))

(defun article-menu-list-metadata-articles ()
  (interactive)
  (article-menu-listing (label-to-propertized-list 'metadata)))

(defun article-menu-list-labels ()
  (interactive)
  (article-menu-listing (label-to-propertized-list 'label)))

(defun list-articles (lis)
  (interactive)
  (article-menu-listing (turn-list-into-propertized-list lis)))
\end{lisp}

\begin{notate}{Using text properties in the article listing} \label{text-properties-in-article-listing}
The approach that I've been using so far, searching for spaces, won't
work if the names of the columns have spaces in them.  Something more
sophisticated (probably using text properties) would work better in
general.  It would also be handy to tuck the articles ``true names''
in as text properties attached to the print names.
\end{notate}

\begin{notate}{On `article-menu-display-article'}
The point is to grab the name of the article on the currrent line of
the listing and display it.  This needs to be checked a bit in the
multicolumn case (which itself needs to be explored).
\end{notate}

\begin{lisp}
(defun article-menu-display-article ()
  (interactive)
  (when (> (line-number-at-pos) 1)
    (save-excursion
      (goto-char (line-beginning-position))
      (search-forward-regexp "[. >] .")
      (setq name-of-current-article
            (get-text-property (point) 'name))
      (article-menu-point-out-current-article)
      (display-article name-of-current-article))))
\end{lisp}

\begin{notate}{On `article-menu-mark-article'}
Use this to mark the article mentioned on this line.
\end{notate}

\begin{lisp}
(defun article-menu-mark-article ()
  (interactive)
  (setq buffer-read-only nil)
  (when (> (line-number-at-pos) 1)
    (goto-char (line-beginning-position))
    (delete-char 1)
    (insert ">")
    (when (< (line-number-at-pos)
             (progn (save-excursion (goto-char (point-max))
                                    (line-number-at-pos))))
      (forward-line 1)))
  (setq buffer-read-only t))
\end{lisp}

\begin{notate}{On `article-menu-unmark-article'}
Use this to remove any mark on the article mentioned on this line.
\end{notate}

\begin{lisp}
(defun article-menu-unmark-article ()
  (interactive)
  (setq buffer-read-only nil)
  (when (and (> (line-number-at-pos) 1)
             (not (save-excursion (goto-char (line-beginning-position))
                                  (looking-at "\\."))))
    (goto-char (line-beginning-position))
    (delete-char 1)
    (insert " ")
    (when (< (line-number-at-pos)
             (progn (save-excursion (goto-char (point-max))
                                    (line-number-at-pos))))
      (forward-line 1)))
  (setq buffer-read-only t))

(defun article-menu-unmark-all-articles ()
  (interactive)
  (setq buffer-read-only nil)
  (save-excursion (goto-line 2)
                  (goto-char (line-beginning-position))
                  (while (re-search-forward "^>" nil t)
                    (replace-match " ")))
  (setq buffer-read-only t))
\end{lisp}

\begin{notate}{On `article-menu-point-out-current-article'}
This is a non-interactive function that puts a dot in front
of the current article when the article listing is generated.
\end{notate}

\begin{lisp}
(defun article-menu-point-out-current-article ()
  (goto-char (point-min))
  (setq buffer-read-only nil)
  (save-excursion (when (search-forward-regexp "^\\." nil t)
                    (replace-match " ")))
  (when (and
         name-of-current-article
         (search-forward-regexp
          ;; maybe this `(format "%s" name-of-current-article)'
          ;; stuff should be stored as some kind of function,
          ;; like `print-name-of-article' or something like that
          (concat "^[. ] " (regexp-quote
                            (format "%s" name-of-current-article)))
          nil t))
    (replace-match (concat ". " (substring (match-string 0) 2))))
  (goto-char (line-beginning-position))
  (setq buffer-read-only t))
\end{lisp}

\begin{notate}{Features of the article menu} \label{features-of-the-article-menu}
The only ``actionable'' feature of the current listing is display.
See also Note \ref{control-panels}.
\end{notate}

\begin{lisp}
(make-generic-menu-mode "Article Menu"
                        '(("g" . article-menu-listing)
                          ("m" . article-menu-mark-article)
                          ("u" . article-menu-unmark-article)
                          ("U" . article-menu-unmark-all-articles)
                          ("q" . quit-window)
                          ("\C-m" . article-menu-display-article)))
\end{lisp}

\begin{notate}{Additional actions}
We also want to have a function that will give the semantics for
``select'' and other sorts of things that we might be able to do to
the objects that we've listed.  (Maybe we should pass this in along
with the objects, and set it up as a local variable that can be called
later.)
\end{notate}

\begin{notate}{Search engine interface} \label{search-engine-interface}
It will be useful to interface this browsing mechanism with a
simple predicate matching mechanism.  Sometimes we'll want to
restrict search (Note \ref{restricting-search}), but for now,
we might as well write a function that searches everything.

A more advanced search engine would apply general predicates
to create the list (e.g. it would allow the user to
restrict to scholia of a certain type, or that \emph{have}
scholia of a given type, or that match other computed properties).

Other basic advances would be to have a second column that
showed the actual match, and to display the matches in order
according to the position of the articles in question within
some specific linearization of the document (e.g. the main one).

The main functions in the ``search engine'' so far are
`article-menu-list-articles-matching-regexp' (Note 
\ref{article-menu-list-articles-matching-regexp}) and
`article-menu-list-articles-matching-predicate' (Note 
\ref{article-menu-list-articles-matching-predicate})
\end{notate}

\begin{notate}{On `article-menu-list-articles-matching-regexp'} \label{article-menu-list-articles-matching-regexp}
This makes a listing showing all of the articles that match
\inp{regexp}.

\emph{(It would be nice if we could leave out meta, reference,
code, and label types of articles in the default version of
this.)}
\end{notate}

\begin{lisp}
(defun article-menu-list-articles-matching-regexp (regexp)
  (interactive "MRegexp: ")
  (let ((matches (mapcar
                  (lambda (name)
                    (propertize (format "%s" name) 'name name))
                  (remove-if
                   (lambda (elt)
                     (not
                      (with-temp-buffer
                        ;; rough, not bothering with
                        ;; `sch-plain-text'
                        (let ((article (get-article elt)))
                          (insert
                           (format "%s"
                                   (scholium-name (get-article elt)))
                           "\n"
                           (format "%s"
                                   (scholium-text article)))
                          (goto-char (point-min))
                          (search-forward-regexp regexp
                                                 nil t)))))
                   (turn-article-table-into-names)))))
    (if matches
        (article-menu-listing matches)
      (message "No hits."))))
\end{lisp}

\begin{notate}{On `article-menu-list-articles-matching-predicate'} \label{article-menu-list-articles-matching-predicate}
This is a way to pick out all of the articles in the library
that match a given predicate.
\end{notate}

\begin{lisp}
(defun article-menu-list-articles-matching-predicate (pred)
  (let ((matches (mapcar
                  (lambda (name)
                    (propertize (format "%s" name) 'name name))
                  (remove-if (lambda (elt) (not (funcall pred elt)))
                             (turn-article-table-into-names)))))
    (if matches
        (article-menu-listing matches)
      (message "No hits."))))
\end{lisp}

\begin{notate}{Restricting search} \label{restricting-search}
If we have lots of subcollections, then we will sometimes need to
search through all of them, and sometimes, only through some subset.
When doing search, there are two efficient ways to limit things (and
these are very similar): either add subcollections that match some
predicate, or remove subcollections that match some predicate.
\end{notate}

\begin{notate}{Saving search results}
It would be handy to be able to store the listings generated by
search in some workspace.  As a prior step, one needs to be able
to turn the arbitrary listing into an article (specifically, a
label)!
\end{notate}

\begin{notate}{Browsing subcollections} \label{browsing-subcollections}
Compare Section \ref{displaying-labels} (should that stuff go here?).
Also note that functionality for displaying sub-objects made out of
hash tables should probably be worked in.  Note that browsing from
list to list is similar to the Gopher experience.

One special sort of subcollection to browse would be one generated by
search.  A variant of `multi-occur' that resolves at the scholium
level would be very handy to have (see Note
\ref{search-engine-interface}).
\end{notate}

\begin{notate}{Display all matches in full}
Instead of just providing the \emph{listing}, sometimes
it could be handy to display a concatenation of all
matching texts (maybe with the matching words highlighted).
Compare Note \ref{derivative-works-in-the-scholium-system}.
\end{notate}

\begin{notate}{On `display-article-listing'}
This just puts the cursor in the article listing.
\end{notate}

\begin{lisp}
(defun display-article-listing ()
  (interactive)
  (pop-to-buffer "*Generic List*"))
\end{lisp}

\begin{notate}{Arrangement of listing window}
The generic list should always go on the \emph{left} in our display,
and similarly with files that have their own buffers.  This is
especially relevant whenever we're treating the listing as an article,
cf. Note \ref{display-label}.  It is possible, although perhaps
unlikely, that sometimes we'll want article listings to be mapped into
the Main Article Display set to a Generic List Mode.
\end{notate}

\begin{notate}{Static catalog display }
It might be advantageous to not have the display disappear -- which
presumably means, display the articles that it lists as scholia
attached to it; otherwise, they display and their scholia display, and
there isn't enough room on the display for the catalog.  Something
like this should be easy enough to do.
\end{notate}

\begin{notate}{Electric catalog}
Probably we could write things so that sections display directly when
the cursor moves over the relevant line in the catalog.  This would
make browsing entries very speedy.
\end{notate}

\begin{notate}{Scholia attached to sections}
When we import the system (Note \ref{import-scholium-system}),
an article is created for each section that is displayed
as a listing.  Such articles are currently displayed
together with all of the articles \emph{in} the section shown
as attached scholia.  It would probably be nice to be able
to turn this ``feature'' off (also perhaps on a 
browsing-style-dependant basis).
\end{notate}

\subsection{Temporal browsing} \label{temporal}

\begin{notate}{History models}
I had the idea to make a generic browser that could be used as
the backend for both the scholium system's temporal browser and
for my lynx-based browser nero.  (It may prove to be most
convenient to simply embed a new nero in the scholium system,
indeed, this seems fairly likely to happen.)  But I think it makes
more sense at this point just to write a working scholia browser.
That sort of abstractification can come later; the design will
feature some effort in this direction, but perhaps largely in the
form of notes about how to make things fully general and
abstract.

What is a temporal browser?  Typically, a list of history elements
that can be added to when new pages are encountered, and from which
elements are deleted as new paths through the network get trod.

For example, if the user browser pages $A$, $B$, and $C$ in order,
there will be a history list $H=(A\ B\ C)$.  Upon going \emph{back},
the history list becomes $H=(A\ B)$, and a future list is created,
$F=(C)$.  If the user then visits a new page $D$, the history list
becomes $H=(A\ B\ D)$ and $F=()$.

In order to make access \emph{quick}, one obvious design strategy is
to store pages that have been browsed locally (i.e., rather than
re-downloading them each time they are browsed).  Of course, if the
pages are already stored locally, this issue is moot.  And in this
case, to save storage space (at a small time cost), we should be
recording \emph{metadata} (names or pointers) on the history list, not
the pages themselves.

There are a couple of other models to consider.

First, we can store our history data in a \emph{tree}.  In this
``\emph{istree}'' model, what we would see in the above scenario is:
\begin{align*}
& T=(A) \\
& T=(A\ B) \\
& T=(A\ (B\ C))) \\
& T=(A\ (B\ C\ D))
\end{align*}

I.e., each level of the tree stores the parent node as the car, and a
list of children as the cdr (children are stored as atoms if they
themselves are childless, or as the head of lists if they have
children).  To keep track of where we are in the tree, we need a list
of directions (cars and cdrs).  In the current case, the list would be
(in the order of application): \emph{cdr, car, cdr, cdr}.  (For
purposes of maintaining a symmetry which will be seen momentarily, we
always return a \emph{list} in which the current node is the car.)  If
the user moves \emph{back} from $D$ to its parent $B$, the
supplementary list becomes \emph{cdr, car}, in other words, we find
the most recent parent (car) in the list.  If the user moves back
again from $B$, the supplementary list becomes \emph{nil}, which means
we are at the first node.

The data structure is simple, but confessedly, the explanation is a
bit complicated!  Another simple model keeps track of the \emph{full
  history}.  In this case, the model evolves as follows:
\begin{align*}
& T=(A) \\
& T=(A\ B) \\
& T=(A\ B\ C\ B\ D)
\end{align*}

This model has the benefit of retaining data about \emph{loops}
(e.g. the short loop $B$, $C$, $B$).  Loops can be lopped off by the
following algorithm: starting at the end of the list (i.e. the current
node), make a list of nodes that have been encountered, and trace
backward.  If the previous node hasn't been encountered, put it at the
beginning of our growing loop-free list.  If it has been encountered,
search backward for the earliest appearance of the last novel node,
and add the element preceding that one to the loop-free list.  Moving
``back'' in the sense of the truncated history list of traditional
browsers can be accomplished by finding antecedent of the earliest
appearance of the current node.  E.g., if the user wants to move
\emph{back}, we add the antecedent $B$ to the list, to obtain
$T=(A\ B\ C\ B\ D\ B)$.  If the user wants to move \emph{back} again, we obtain
$T=(A\ B\ C\ B\ D\ B\ A)$, since $A$ is the earliest immediate
prequel to $B$.
Similarly, to move \emph{forward} at this point, we find the
latest immediate sequel to the current node; in this case, $B$ again,
and put $T=(A\ B\ C\ B\ D\ B\ A\ B)$.

We can also recover the data stored in the istree model. This is left
as an easy exercise to the reader, but the algorithm is given in the
code; for reasons of generality (and simplicity!), \emph{we are
  adopting the full history model as the basis of this browser}.

A variety of interesting novel browser actions can be performed
using this information, e.g., finding all the pages you looked at
before or after this one.
\end{notate}

\begin{notate}{Science (fiction) tie-in}
Users can tap into the lived experience of a time lord (assuming we
retain the ``temporal'' metaphor), or the experience of a Wolframian
particle (compare NKS) if we consider a spatial metaphor instead.
We'll let you sort out the paradoxes.  (\emph{Hint:} you never
step in the same river twice.)
\end{notate}

\begin{notate}{Tabbed browsing}
Users of the full history model may be in less immediate need of
``tabs'' (in the form of multiple histories) than users of the truncated
history model.  Alternate browsing paths are (close to) natively
available.  Futhermore, a page can be marked as a divider between
one ``tab'' and another.  Indeed, this gives us an analogy to the
``tabs'' than stick out of file folders in a physical filing cabinet.
(The only difference is that pages can be in multiple files at the
same time, which is as it should be.) This means that
multiple histories can be used (in a read-only mode) as virtual
\emph{cabinets} for organizing information.  The actual system of
cabinetry need not be maintained here; implementing extra browsing
features is a good activity for some rainy day.

On the other hand, see Note \ref{thematic-histories}.
\end{notate}

\begin{notate}{Thematic histories} \label{thematic-histories}
It might be useful to keep a record of several different sorts of
histories, e.g. of editing events or ``catalog pages'' (Note
\ref{history-for-listing}).  These
alternate histories wouldn't have much to do with the temporal
browser per se.  Emacs does this in some cases (e.g. recording the
input history for various interactive functions separately).

If we're going to do this, probably the code in this section
should be written in a somewhat more generic way.
\end{notate}

\begin{lisp}
(defvar sb-history nil)

(add-hook 'display-article-hook
          (lambda ()
            (sb-add-article-to-history name-of-current-article)))
\end{lisp}

\begin{notate}{Instead of adding the name} \label{instead-of-names}
Maybe we should be adding directions on how to get the article.
The name might not be sufficient if we end up working with
nested structures.
\end{notate}

\begin{notate}{Complexities could be handled by `display-article'}
Of course, additional sorts of access instructions (like those alluded
to in Note \ref{instead-of-names}) could be handled by
`display-article'.  \emph{This section} does not depend on article
names being stored as strings.
\end{notate}

\begin{lisp}
(defun sb-add-article-to-history (article)
  (setq sb-history (nconc sb-history (list article))))

(defun sb-back ()
  (interactive)
  (let ((current (car (last sb-history)))
        (n 0)
        found)
    (while (not found)
      (if (equal (nth n sb-history) current)
          (progn (setq found t)
                 (if (> n 0)
                     (display-article (nth (1- n) sb-history))
                   (message "Already at beginning of history.")))
        (setq n (1+ n))))))

(defun sb-forward ()
  (interactive)
  (let* ((current (car (last sb-history)))
         (max (1- (length sb-history)))
         (n (1- max))
         found)
    (while (not found)
      (if (equal (nth n sb-history) current)
          (progn (setq found t)
                 (if (< n (1- max))
                     (display-article (nth (1+ n) sb-history))
                   (message "Already at end of future")))
        (setq n (1- n))))))
\end{lisp}

\begin{notate}{Keeping track of visited references} \label{visited-references}
It seems like we might want to keep track of information about visited
references in some data structures maintained by the temporal browser.
Alternatively, we could put this information into the metadata
articles attached to the articles that have been visited (which would
make it easy to report the number of visits to the article owners).
See Note \ref{reference-face}.
\end{notate}

\begin{notate}{Function to go all the way back}
Should be able to go all the way back to the beginning
of the history list in one go.
\end{notate}

\begin{notate}{On `sb-previous'}
This gives the most recently browsed article besides the current
one.
\end{notate}

\begin{lisp}
(defun sb-previous ()
  (car (last sb-history 2)))
\end{lisp}

\begin{notate}{Be careful to avoid cycles}
I'm not sure that the formulation of temporal browsing given in this
section actually works!  It needs a critical review to make sure that
the future actually has an end, for example.  I seem to be discovering
problems by using display-only text (see Note
\ref{display-purposes-only}).
\end{notate}

\subsection{Linear browsing} \label{linear-browsing}

\begin{notate}{Browsing through the contents of a label} \label{browsing-through-the-contents-of-a-label}
In Note \ref{multidimensional-visual-browsing}, the idea of being able
to move forward through the document's natural structure was
discussed.  One of the easiest ways to define this sort of structure
is relative to the contents of some label. A more advanced mode would
stitch together several labels, e.g., the sections and subsections of
a document, with some simple logic for moving from label to label.  In
addition, a given article might appear as part of several labels, in
which case, the user could be presented with possible directions to
travel in.  This is in some sense an extension of Section \ref{local},
since it gives more local directions to go in.  It is also similar to
the idea of following through a set of slides (and presumably would
make it easy to implement such an arrangement).

The key thing seems to be identification of the ``parent'' label(s)
to use for this purpose.  Maybe this info is already available in
the form of backlinks?  Although, note that we might not always
want to turn on this sort of label browsing for all labels all the
time.
\end{notate}

\begin{notate}{Disposition of code for linear browsing}
Some code for additional graphical features has been added
to the scholia display by `scholia-display-extras'.  It would
be good to add some commands to do a semantically similar
sort of navigation non-graphically, here.  When the multi-parent
browsing ideas from Note \ref{more-than-one-parent} have
become fixed, they should of course be available here too.
\end{notate}

\begin{notate}{On `add-visible-back-temporal-link'} \label{add-visible-back-temporal-link}
This makes a link to the previous-browsed page appear in the Scholia
Display.  It runs within the scope of `scholia-display-extras' (Note
\ref{scholia-display-extras}).

It like we should potentially be able to change the strategy here, so
as to add more information to the text properties and not actually
create any corresponding article.  I think this would require some
adjustments to the way text properties are handled.
\end{notate}

\begin{lisp}
(defun add-visible-back-temporal-link ()
  ;; this should be marked up to become a fake link (one that isn't
  ;; displayed in the main buffer)
  (when (sb-previous)
    (insert "\n\n"
            "Back (temporal): ")
    (let ((beg (point)))
      (insert (propertize 
               (format "%s" (sb-previous)) 'face 'sch-reference-face))
      (scholium 'reference-to-previous-article
                nil
                `((,name-of-current-article
                    (passage 
                     ,beg
                     ,(point)))
                  (,(sb-previous)))
                '(reference fake))
      (add-to-scholia-property-within-region
       beg
       (point)
       '(reference-to-previous-article 1 1)))))
\end{lisp}

\begin{notate}{On `add-visible-parent-and-sibling-links'} \label{add-visible-parent-and-sibling-links}
This adds links to the parent and nearest siblings in the Scholia
Display buffer.  It runs within the scope of `scholia-display-extras'
(Note \ref{scholia-display-extras}).
\end{notate}

\begin{lisp}
(defun add-visible-parent-and-sibling-links ()
  ;; identify the link to the parent, if it exists.
  ;; (this assumes that there is only one parent)
  (let ((link-to-parent (car (member-if (lambda (link)
                                          (member 'parent (cdr link)))
                                         (scholium-about article)))))
    (when link-to-parent
      (let* ((parent (get-article (first link-to-parent)))
             (parent-data
              (scholium-text parent))
             (this-name-headed (member name-of-current-article
                                       parent-data))
             (next (cadr this-name-headed))
             (prev (car (last (butlast parent-data
                                       (length this-name-headed))))))
        (when parent
          (insert "\n\n"
                  "Parent: ")
          (let ((beg (point)))
            (insert (propertize (format "%s" (scholium-name parent))
                                'face 'sch-reference-face))
            (scholium 'reference-to-parent
                      nil
                      `((,name-of-current-article
                          (passage 
                           ,beg
                           ,(point)))
                        (,(scholium-name parent)))
                      '(reference fake))
            (add-to-scholia-property-within-region
             beg
             (point)
             '(reference-to-parent 1 1))))
        (when prev
          (insert "\n\n"
                  "Back (in parent): ")
          (let ((beg (point)))
            (insert (propertize (format "%s" prev) 'face 'sch-reference-face))
            (scholium 'reference-to-previous-article-in-parent
                      nil
                      `((,name-of-current-article
                          (passage 
                           ,beg
                           ,(point)))
                        (,prev))
                      '(reference fake))
            (add-to-scholia-property-within-region
             beg
             (point)
             '(reference-to-previous-article-in-parent 1 1))))
        (when next
          (insert "\n\n"
                  "Forward (in parent): ")
          (let ((beg (point)))
            (insert (propertize (format "%s" next)
                                'face 
                                'sch-reference-face))
            (scholium 'reference-to-next-article-in-parent
                      nil
                      `((,name-of-current-article
                          (passage 
                           ,beg
                           ,(point)))
                        (,next))
                      '(reference fake))
            (add-to-scholia-property-within-region
             beg
             (point)
             '(reference-to-next-article-in-parent 1 1))))))))

(add-hook 'scholia-display-extras-hook
          'add-visible-back-temporal-link)
(add-hook 'scholia-display-extras-hook
          'add-visible-parent-and-sibling-links)
\end{lisp}

\begin{notate}{On `forward-in-parent'} \label{forward-in-parent}
A quick command to get the label identified as this object's parent,
and move to the next mentioned after this one in this label.  (It
would also be possible to follow the next reference in a page, i.e.,
use a page like a label; but this function doesn't do that.)

It would be nice to have this function select the ``next cousin'' (or
other suitable relative) if we are out of siblings.

We'll also want a quick command to hop \emph{to} an article's parent.
\end{notate}

\begin{lisp}
(defun forward-in-parent ()
  (interactive))
\end{lisp}

\begin{notate}{On `backward-in-parent'} \label{backward-in-parent}
Like `forward-in-parent' (Note \ref{forward-in-parent}) but moves
backward instead.
\end{notate}

\begin{lisp}
(defun backward-in-parent ()
  (interactive))
\end{lisp}

\section{Editing and deleting} \label{editing}

\subsection{Initiating edits} \label{initiating-edits}

\begin{notate}{How to initiate edits}
Buffers are frequently non-read-only; if these buffers have been
marked up with scholia, then they can be edited directly.  The Main
Article Display buffer (for scholia with no other or better home) may
as well be non-read-only as well.  The point being that to start
editing the \inp{text} of an article, one just needs to begin.  (In
the current implementation, articles in the Scholia Display buffer
need to be made current with `follow-scholium', or by some other
means, but that is easy enough.)

However, the other parts of an article (Note
\ref{definition-of-scholium}) are not immediately editable without
some intermediate step to call them up and expose them to the user.
Since \inp{bookkeeping} data is maintained by the system itself, the
only parts of articles that need special initial steps to edit are the
\inp{name}, \inp{about}, and \inp{type} fields.
\end{notate}

\begin{notate}{Inserting text in the middle of a marked up region} \label{inserting-text-in-middle-of-marked-up-region}
When someone inserts text into the middle of a marked up region, does
the new text take on the markup properties of the surrounding text, or
does it not, or can we make it an option, to possibly be exercised in
different ways in different places?  I assume we will want different
behavior at different times.  (Cf. Note
\ref{splitting-text-properties}).  The reference to consider is the
``Sticky Properties'' node from the Elisp manual\footnote{(info
  "(elisp)Sticky Properties")}.

There they tell us that
\begin{quotation}
By default, a text property is rear-sticky but not front-sticky; thus,
the default is to inherit all the properties of the preceding
character, and nothing from the following character.
\end{quotation}
Furthermore,
\begin{quotation}
If a character's `rear-nonsticky' property is `t', then none of its
properties are rear-sticky.  If the `rear-nonsticky' property is a
list, properties are rear-sticky \emph{unless} their names are in the
list.
\end{quotation}
Thus, we have nothing to worry about in terms of front-stickiness, but
we must maintain an appropriate rear-nonsticky list throughout the
regions in which we want some of the properties to be non-sticky.

Something like this (enhanced, of course) will work:
\begin{idea}
(progn (put-text-property (line-beginning-position) (line-end-position)
       'face 'italic)
       (put-text-property (line-beginning-position) (line-end-position)
       'rear-nonsticky '(face)))
\end{idea}
\end{notate}

\subsection{Editing `about' data} \label{editing-about-data}

\begin{notate}{The joys and sorrows of editing `about' data}
As noted in \ref{consistency-editing}, changes to \inp{about} data can
have broad ramifications.  In addition to masking any explicit links
that point at this data, we will have to change everything that uses
link-id's. 

In particular, backlinks corresponding to the set of edited links will
have to change.  I think that this part at least can be accomplished
by running `scholium-modified-about-hook' (cf. Note
\ref{hooks-for-new-and-modified-scholia} and Note
\ref{put-backlinks}).

We'll have to decide whether it is worth our trouble to change markup
(it would seem to be inavoidable; however, as a quick work-around,
perhaps other renderings can simply be blanked out when \inp{about}
data is being edited).
\end{notate}

\subsection{Finding revised `about' data by parsing text properties} \label{parsing}

\begin{notate}{Mini-overview of parsing text properties}
This section concerns the topic of finding changes to \inp{about} data
of scholia that are induced by changes to the \inp{text} of an article
that they are \inp{about}.
\end{notate}

\begin{notate}{Overview of parsing text-properties} \label{overview-of-parsing-text-properties}
By finding the places in a marked-up buffer where the `scholia'
property changes and examining its value before and after the change
point, the beginning and end of each marked region can be found.  (See
the description of this property in Note \ref{the-scholia-property}.)

More specifically: (1) we must find all the places where the `scholia'
property changes; (2) then figure out what was new or what went away
at each change point; (3) and update a list of beginnings and endings
in the approximate form of an \inp{about} list.  This is a first step
in the lengthier process of committing edits (see Section
\ref{committing}).
\end{notate}

\begin{notate}{Indicator functions} \label{indicator-functions}
We can have marked regions that overlap to any depth; overlapping
scholia can also have non-trivial non-overlapping regions.

In a mathematical abstraction, we might see scholia like
$\indicator{[0,3]}$, $\chi_{[1,10]}'$, $\chi_{[2,5]}''$,
$\chi_{[8,9]}'''$ (as pictured in Figure
\ref{Simulation-with-indicator-functions}).  We could even have
different regions from the same scholium overlap.

It would be nice to be able to indicate arbitrary subsets of the
\inp{text} that we're presented with.  The actual set of available
indicators is a $\sigma$-algebra, if I remember correctly.

For LISP code, picking out arbitrary bits of list structure would be
handy (see Note \ref{including-transcluding-and-lisp}).
\end{notate}

\begin{notate}{On `scholia-property-at-changepoints'}
This returns a list of pairs; points where
the `scholia' property changes paired with the value of the `scholia'
property at those points.

\begin{figure}
\begin{center}
%% in LaTeX, ascii art is a little bit trickier than usual
{\tt
\begin{tabular}{l}
\ \ \ \ \ \ \ \ -- \\
\ \ ------ \\
\ ------------------  \\
------  \\
1234567890
\end{tabular}}
\end{center}
\caption{Markup simulated with indicator functions \label{Simulation-with-indicator-functions}}
\end{figure}
\end{notate}

\begin{lisp}
(defun scholia-property-at-changepoints ()
  (let ((next-change-point (point-min))
        change-points)
    (while next-change-point
      (setq change-points (cons (list next-change-point
                                      (get-text-property
                                       next-change-point
                                       'scholia)) 
                                change-points))
      (setq next-change-point (next-single-property-change+predicate
                               next-change-point
                               'scholia
                               'equal)))
    (when (not (null (get-text-property
                      (caar change-points)
                      'scholia)))
      (setq change-points (cons (point-max) change-points)))
    (reverse change-points)))
\end{lisp}

\begin{notate}{On `detect-scholia-structure'} \label{detect-scholia-structure}
This function says which passages of the current article have been
marked up with which scholia.  Typically, this function will be run
after editing, to recover the modified markup information and
propagate it to attached scholia (see Section \ref{committing}).

The way the function works is as follows.  At each of the change
points, we find the the annotations that have either just appeared or
just disappeared, by using the `set-difference' function to compare
the value of the `scholia' property at the \emph{upcoming} change
point with the value at the \emph{current} one.  We then sort the
elements of this difference into ``opening'' and ``closing'' sets
based on whether they are present at the \emph{upcoming} change point,
or the \emph{current} one.  In order to ensure that there is an
adequate basis for comparison in all cases, exogenous points at
``$-\infty$'' and ``$\infty$'' are introduced, with no scholia
attached to them.

Every ``opening'' element corresponds to some as-yet-unseen marked-up
region; when encountering such an item, we add a term to the return
value that looks like ``(VALUE BEG)'' where VALUE and BEG are,
respectively
\begin{itemize}
\item the value of the `scholia' property -- which will be the
 name of a scholium together with a unique link identifier, as
 described in Note \ref{the-scholia-property};
\item and the point where this link starts to attach to the current
 article.
\end{itemize}
Similarly, every ``closing'' element corresponds to the end of some
already-seen marked-up region.  When encountering such elements, we
modify the corresponding previously added term in the return value to
read ``(VALUE BEG END)'', END here being the point where the link in
question stops attaching to the current document.

Note that since `scholia-property-at-changepoints' gives us the values
in increasing order, new ``opening'' items are encountered and added
to the list in increasing order as well.  This property is important
for later.
\end{notate}

\begin{lisp}
(defun detect-scholia-structure ()
  (let ((begs-and-ends
         (cons '(-infinity nil)
               (append
                (scholia-property-at-changepoints)
                (list '(infinity nil)))))
        marked-regions)
    (while (> (length begs-and-ends) 1)
      (let ((difference (set-difference (second
                                         (first begs-and-ends))
                                        (second
                                         (second begs-and-ends))))
            opening
            closing)
        (when difference
          (dolist (elt difference)
            (if (member elt (second (second begs-and-ends)))
                (setq opening (cons elt opening))
              (setq closing (cons elt closing))))
          (dolist (elt opening)
            (setq marked-regions
                  (add-to-or-start-list
                   marked-regions
                   `(,elt ,(first (second begs-and-ends))))))
          (dolist (elt closing)
            (let ((found (member-if (lambda (pass)
                                      (equal (first pass) elt))
                                    marked-regions)))
              (setcdr (cdr (car found))
                      (list (first (second begs-and-ends)))))))
        (setq begs-and-ends (cdr begs-and-ends))))
    marked-regions))
\end{lisp}

\begin{notate}{New scholia added while editing}
I don't \emph{think} that new scholia added while editing will need to
be treated any differently during the reparsing process from scholia
that were created before editing started.  Scholium-creation should be
a self-contained process that gets everything stored properly; the
only special concern is that text properties would need to be added to
the buffer that is being edited, but that's obvious enough.
\end{notate}

\begin{notate}{Finding changes in scholia about the whole article} \label{changes-in-scholia-about-whole-article}
Presumably this data will be stored in a relevant metadata article,
rather than in text properties (contrast Aaron's picture).  We
probably won't have to do reparsing to handle scholia like this,
\emph{but we will have to do something}.
\end{notate}

\begin{notate}{Make `scholia' a variable?}
It would be easy enough to make `scholia' as used above
into a variable instead of hard code, so that we could read from
another text property instead.  Probably we would want to complement
this with similar changes to the code that adds text properties!
\end{notate}

\subsection{Committing edits} \label{committing}

\begin{notate}{Themes for committing}
(1) Store stuff to metadata articles;
(2) Prefer these for rendering purposes.
\end{notate}

\begin{notate}{Committing implications}
Edits to any part of a scholium (in particular, to its \inp{name},
\inp{text}, \inp{about}, or \inp{type} data) need to be saved and
otherwise processed.  Sometimes changes in one article will result in
changes to other articles, e.g., \emph{changing \inp{about} data may
  lead to changes in backlinks}; and changes to \inp{text} may lead to
changes in one or more attached scholium's \inp{about} data (Note
\ref{set-updated-about-info}).
\end{notate}

\begin{notate}{Bookkeeping and version managing}
When articles have been modified, \inp{bookkeeping} data needs to be
updated.  In particular, we are supposed save sufficient data to move
back and forth between article versions.  This is true of both
\inp{text} data and the other data that make up articles (\inp{name}
may be handled somewhat differently, but in general we need to store
enough data to move between all versions of the article).
\end{notate}

\begin{notate}{Setting updated `about' information} \label{set-updated-about-info}
The idea of preferring metadata articles when rendering has been
discussed in the section on metadata articles (Section
\ref{metadata-articles} and Note \ref{updating-link-extent} in
particular), and is reflected in the way `mark-things-up' is written
(Note \ref{mark-things-up}).  For this all to work, it is necessary
for `commit-edits' to store updated \inp{about} data when it runs.
\end{notate}

\begin{notate}{On `store-link-masks'} \label{store-link-masks}
This function runs within the scope of `commit-edits', after
`adjusted-markup' has been found by parsing (Note \ref{commit-edits}).
Indeed, so far, this is the only significant default action of
`commit-edits' (cf. Note \ref{set-updated-about-info}).

The function compares the elements of `current-markup' (Note
\ref{current-markup}) to the elements of `adjusted-markup'.  When
changes are present in the markup data, `store-link-masks' adds or
changes masks.

More precisely, for each element of `current-markup', we select
from `adjusted-markup' those elements with the same link-id
(or maybe mask-id).   These are the `matching-regions' corresponding
to that markup element.  Unless there is only one matching region
and this matching region happen to point to extend over exactly
the range specified by the original link, we execute an operation
on masks.

This operation is to swap in a replacement mask, if the markup
element was coming from a mask already, or to create a brand new
mask otherwise.  If we're doing a replacement, we look for the
\emph{one} mask that corresponds to the link-id borne by the
current markup element.  We then substitute \emph{all} of the
matching regions, for whatever was previously stored in this mask.

Otherwise, we just create a new mask with the appropriate link-id,
containing the relevant regions.
\end{notate}

\begin{lisp}
(defun store-link-masks ()
  (dolist (link-ext current-markup)
    (let* ((matching-regions (remove-if
                              (lambda (markup-item)
                                (not (equal (car markup-item)
                                            (car link-ext))))
                              adjusted-markup))
           (num-matching-regions (length matching-regions))
           (name-of-linking-article (if (eq (first
                                             (car link-ext)) 'mask)
                                        (first (second
                                                (first link-ext)))
                                      (first (first link-ext)))))
      (unless (and (eq num-matching-regions 1)
                   (equal (cdr (first matching-regions))
                          (cdr link-ext)))
        (if (eq (caar link-ext) 'mask)
            ;; swap in a different mask
            (let* ((masks
                    (get-metadata-field 'masks
                                        name-of-linking-article))
                   (current-mask-headed
                    (member-if (lambda (item)
                                 (equal (second (car link-ext))
                                        (car item)))
                               masks)))
              (setcdr (car current-mask-headed)
                      (mapcar (lambda (ided-reg) 
                                (cdr ided-reg))
                              matching-regions))
              (put-metadata-field 'masks
                                  masks
                                  name-of-linking-article))
          ;; create a new mask
          (let ((masks (or (get-metadata-field
                            'masks
                            name-of-linking-article)
                           (list 'masks))))
            (put-metadata-field
             'masks
             (setcdr
              masks
              (add-to-or-start-list
               (cdr masks)
               `(,(car link-ext)
                 ,@(mapcar (lambda (ided-reg) (cdr ided-reg))
                           matching-regions))))
             name-of-linking-article)))))))
\end{lisp}

\begin{notate}{Adding new markup to rendered articles} \label{adding-new-markup-to-rendered-articles}
Adding new markup to rendered articles
will introduce a discrepancy
between `adjusted-markup' and `current-markup', i.e., items
will be found that weren't there before.  This shouldn't,
be a problem, because `store-link-masks' iterates through the
members of `current-markup' looking for matches.  So, anything
completely new in `adjusted-markup' will be ignored by
the `store-link-masks' phase of the commit process.  This
is relevant to Section \ref{derivative}, and any other code
that modifies the markup of rendered articles.
\end{notate}

\begin{notate}{On `commit-edits-hook'} \label{commit-edits-hook}
This is the single point of customization for the commit function,
to be used in future sections.  It runs within the scope of
`commit-edits', after `adjusted-markup' has been found
by parsing.  Thus, functions that are added to this hook can
take advantage of this knowledge about the current state of
markup.
\end{notate}

\begin{lisp}
(defvar commit-edits-hook nil)
\end{lisp}

\begin{notate}{On `after-committing-edits-hook'} \label{after-committing-edits-hook}
This facilitates additional actions after the main committing
routines are finished.
\end{notate}

\begin{lisp}
(defvar after-committing-edits-hook nil)
\end{lisp}

\begin{notate}{Summary of idea behind `commit-edits'}
When you edit the buffer, the text properties showing where scholia
are attached move around.  When you run `commit-edits', metadata
indicating the new position of these regions is stored; it will be
used subsequently so that the correct regions are highlighted.  (These
suggestions coming can later be incorporated into the scholia by their
authors.)  Furthermore, `commit-edits' stores a version of the article
for subsequent browsing.
\end{notate}

\begin{notate}{On `commit-edits'} \label{commit-edits}
Anything run by the `commit-edits-hook' that changes the text of the
article being committed should store the desired value on
`adjusted-text'.  It is going to take a bit of thinking about how to
make several functions (particularly functions having to do with
derivatives) run properly, in serial.  (But I'm pretty sure it can be
done.)

Note that when committing we currently make sure that exactly one
newline is attached to the end of the page when the text is
saved.  This has a beneficial effect on exporting the scholium
system (see \ref{export-scholium-system}).

We may at some point want one or more commit functions that will store
more different kinds of edits from more different places.  For
example, we'll want to be able to commit edits that take place in the
Scholia Display buffer.  Similarly for edits to `about' data, and so
forth (see Note \ref{editing-about-data}).
\end{notate}

\begin{lisp}
(defun commit-edits ()
  (interactive)
  (let* ((adjusted-markup (copy-tree (detect-scholia-structure)))
         (old-contents (get-article name-of-current-article))
         (adjusted-text (buffer-substring-no-properties (point-min)
                                                        (point-max)))
         (old-text (scholium-text old-contents)))
    (store-link-masks)
    (run-hooks 'commit-edits-hook)
    ;; here is where the criterion for checking that some change has
    ;; actually been made would be inserted
    (when (not (equal adjusted-text old-text))
      (scholium name-of-current-article
                (with-temp-buffer (insert adjusted-text)
                                  (goto-char (point-max))
                                  (insert "\n")
                                  (delete-blank-lines)
                                  (buffer-substring-no-properties
                                   (point-min)
                                   (point-max)))
                ;; of course, for more advanced versions of
                ;; this code, these won't be static
                (scholium-about old-contents)
                (scholium-type old-contents)
                ;; this should be reformatted to store the old
                ;; version!
                (scholium-bookkeeping old-contents)))
    (run-hooks 'after-committing-edits-hook)))
\end{lisp}

\begin{notate}{After committing, redisplay article}
Hopefully this won't cause any problems.  The one concerning point I
can think of might come up when committing fully rendered and
identified compilations (Section \ref{quick-compilations}), but we
haven't worked out the major details of that yet, so there's no reason
to worry about this one minor one.
\end{notate}

\begin{lisp}
(add-hook 'after-committing-edits-hook 'redisplay-article)
\end{lisp}

\begin{notate}{Shortcomings of the basic commit implementation}
Old comments should continue to apply verbatim to the old version of
the document -- and also, tentatively, to the new version.  Maybe
masks should contain information about which version of the article
they apply to.  A given link could have a collection of masks, each
relevant to a different version of the article being linked to
(perhaps recycling content when the masks are known to be the same).

And, of course, commiting is going to have to store a version of the
article on the \inp{bookkeeping} list (or equivalent; see Note
\ref{vc-and-filesystem-support}).  We will then want support for
\emph{version browsing} -- hopefully the browsing functions can be
made to ``just work'' on data coming in with an explicit version
number.

And, in addition, we should be updating backlinks as needed.

(These items should be assessed to see how critical they are; it seems
to me that we can probably work with things roughly as they are, if we
aren't doing anything too wild.  It would be interesting to document
the process of coming to rely more heavily on the system as time goes
by.)

Another nice thing would be to make sure that there is a
newline at the end of the committed text.
\end{notate}

\begin{notate}{Assigning masks after reparsing} \label{send-messages-to-redirected-scholia}
We send a message to each of the scholia that have been redirected
after editing has taken place (cf. Note \ref{masks}).  In order to do
this, we have to compare the new markup data with the old markup data
to figure out what will change.  We may want to keep a buffer-local
record of the original markup data in order to figure out precisely
what has changed.  (I think for now we just assume that there is one
rendered article at a time, so we only keep track of one set of old
markup data.  But don't quote me on that.)

Alternatively, we could mask \emph{everything}, regardless of whether
or not this data has actually changed, but this seems rather silly,
and potentially very inefficient besides.  So we don't do that.
\end{notate}

\begin{notate}{Similarity of deleting and committing} \label{similarity-of-deleting-and-committing}
Note that deleting is probably going to be similar to committing, in
terms of the way changes to \inp{about} data are reported and stored.
(See Section \ref{deletion}.)
\end{notate}

\begin{notate}{Where to store committed edits}
One thing I'm somewhat curious about is whether it is ever safe to
assume that we can make changes to an attached scholium
\emph{directly}, without waiting for its owner's input.  It seems more
likely that we would want to put the updated information on that
scholium's \emph{metadata article}, which is its ``world-writable''
space (see Note \ref{metadata-article-overview}).  If we do this,
that means that the routine we use to find a scholium's \inp{about}
data when rendering a given article may have to look at the scholium's
metadata article.  (The case in which the same person owns both
articles might be one case in which we would wish to make changes
directly.)
\end{notate}

\begin{notate}{The role of metadata} \label{role-of-metadata}
If we assume that changes aren't made directly, then the owner of the
scholium whose metadata article has been changed will have the
opportunity to accept or decline the suggested changes.  Note that if
the author doesn't accept the changes, then the rest of the world may
still have the option to use \inp{about} data from the metadata
article instead of from the article itself.  Also note that even if a
commentator retracts remarks for some updated version of the article,
they may continue to apply to earlier versions.

When a commentator accepts changes, most likely what will happen is
that a new version of the scholium will be created, and the specific
suggestions removed from its metadata article.  However, if the
suggestion is declined, most likely the suggestions will stay on the
metadata article.  (This situation seems a bit backwards.)
Alternatively, if the suggestions are declined, the author might just
delete them (if we make it so that anyone can make changes to the
metadata article -- which might be reasonable).

For more on this topic, see Section
\ref{responding-to-incoming-metadata}.
\end{notate}

\begin{notate}{The possibility for conflicts in backpropagated `about' data} \label{conflicts-in-masks}
The potential for conflicts in suggested changes coming from edits to
various different articles exists, in theory, but I think it is
dispatched by the fact that each region is drawn from exactly one
article. So, as long as each link is kept up-to-date with respect to
any changes in the one article that it applies to, there won't be any
conflict.
\end{notate}

\begin{notate}{Conflicts in a distributed environment} \label{conflicts-in-a-distributed-environment}
Of course it is the case that if the same article is being changed or
even just talked about in several different places, or if it is a
scholium that applies to several articles that reside in different
repositories, many different sorts of conflicts could arise.  The task
of assimilating changes from various locations should be given at
least partial support in code.
\end{notate}

\begin{notate}{Committing and reverting}
Note that ``revert'' is in some weak sense an opposite action to
``commit''.  (See also Note
\ref{similarity-of-deleting-and-committing}.)
\end{notate}

\subsection{Editing en masse} \label{masse}

\begin{notate}{Enhanced control panel}
It might be good to add another column to the article listing control
panel.  Any changes to the article listing that take place in this
section should be done somewhat delicately and artistically.
Presumably this implies that the list of column-fields should be
stored as a variable, so that it can be easily modified.  Note that
functions for mapping over lists of articles forms the theoretical
basis of this section (see Note
\ref{mapping-across-a-subcollection}).
\end{notate}

\begin{notate}{Adding metadata en masse}
It isn't quite ``editing'', but equally important would
be a way to apply a given scholium to a set of articles
(both severally and not severally), or add type data to
a set of articles that you own, or whatever.
\end{notate}

\begin{notate}{On `label-marked-articles'}
Apply the specified \inp{label} to each marked article.
New labels can be applied no problem.
\end{notate}

\begin{lisp}
(defun label-marked-articles (label)
  (interactive (list
                (intern (let ((completion-ignore-case t))
                          (completing-read
                           "Label: "
                           (label-to-list 'label))))))
  (let (article-names)
    (save-excursion
      (goto-line 2)
      (goto-char (line-beginning-position))
      (while (re-search-forward "^> ." nil t)
        (setq article-names
              (append article-names
                      (list
                       (get-text-property (point) 'name))))))
    (dolist (name article-names)
      (label-article name label))))
\end{lisp}

\subsection{Deletion} \label{deletion}

\begin{notate}{Deleting articles} \label{deleting-articles}
It is sometimes desirable to delete articles that are no longer worth
looking at.  We want to give the user this power, though in some cases
it would be nice if the article wasn't really deleted and was instead
just \emph{depreciated}.

There is a somewhat complex operation that should be executed when an
article is actual deleted (and/or, conversely, a somewhat complex
system we should put in place to deal, at render time, with articles
that have been deleted without sufficient follow-through).  Some of
the steps in the deletion process should be: (1) deleting backlinks;
(2) unindexing the article from whatever labels the article has been
indexed in.

But right now, we just delete.
\end{notate}

\begin{notate}{Steps to be taken when deleting an article} \label{steps-when-deleting}
If the article is a scholium attached to the current article,
we should remove the associated text properties and overlays.

Since in theory we can't modify the \inp{about} data that point
at the deleted article, we might want to broadcast a signal to
these articles that say that their referent has been deleted.
One way to do this would be to replace the article with a
``Deleted Article'' stub, and probably add it to a ``deleted''
label.

Of course, as with other issues having to do with \emph{propagation}
things are somewhat different if all the articles are owned
by the same person (or otherwise have conventions associated
with them whereby they can be kept in sync without problems).
\end{notate}

\begin{notate}{Set current article to `nil'}
If the current article is deleted, then the `name-of-current-article'
variable should probably be set to nil.  (Either that, or something
from the temporal browser should be used to set the variable to the
last-browsed page.)
\end{notate}

\begin{lisp}
(defvar delete-article-hook nil)

(defun delete-article (name)
  (interactive (list (read-article-name)))
  (let ((article (get-article name)))
    (if article
        (progn
          (remhash name article-table)
          (when (equal name name-of-current-article)
            (setq name-of-current-article nil))
          (run-hooks 'delete-article-hook))
      (error "No article by that name found"))))
\end{lisp}

\begin{notate}{Deal with effects on temporal browser}
It is a little weird to revised history to make it look like deleted
pages \emph{never existed}.  Seriously Orwellian.  For now, that's
exactly what we do.
\end{notate}

\begin{notate}{On `remove-appearances-in-history'}
This function will run inside the scope of `delete-article'.
\end{notate}

\begin{lisp}
(defun remove-appearances-in-history ()
  (setq sb-history (delete name sb-history)))

(add-hook 'delete-article-hook 'remove-appearances-in-history)
\end{lisp}

\begin{notate}{Deal with effects on articles that deleted article was attached to}
Presumably articles that the deleted article linked to will have to
have their backlinks adjusted.  That is to say, it would be somewhat
odd to have backlinks to an article that doesn't exist.
\end{notate}

\begin{notate}{On `delete-scholium-associated-with-current-marked-region'}
Provides a quick way to delete the scholium attached to region
containing point.  If more than one scholium is so attached,
we bring up a menu to select items for deletion.

This function assumes that the `scholia' property is simply
comprised of a list of the names of attached scholia; I'm
not so sure that this is an accurate assumption (if not,
then we should be able to easily extract such a list from
the actual `scholia' property).

This is just one of several functions that would benefit from
the addition of progressive markup modifications.
\end{notate}

\begin{lisp}
(defun delete-scholium-associated-with-current-marked-region ()
  (interactive)
  (let ((linked-scholia (scholia-named-at-point)))
    (cond
     ((eq (length linked-scholia) 0)
      (message "No scholium attached here."))
     ((eq (length linked-scholia) 1)
      (delete-article (car linked-scholia)))
     (t
      (list-articles linked-scholia)
      (message
       "Inspect articles and select items for deletion with \"d\".")))))
\end{lisp}

\begin{notate}{Deletion in various strengths}
Probably there should be several different kinds of
deletion (like in X windows, there is kill, destroy, minimize, wipe
out to the fifth generation, etc.; also, compare the several
different ``strengths'' derivative articles come in).
\end{notate}

\subsection{Responding to incoming metadata} \label{responding-to-incoming-metadata}

\begin{notate}{How to deal with metadata}
This section gives support for article owners dealing with incoming
metadata coming from editing events.  An important example: scholia
may see suggested changes to their \inp{about} data when the
article(s) that they are about change).  Authors might want to
incorporate such incoming changes into their document, or obliterate
them, or revise them.  Perhaps a scholium author will want their
comments to apply to the version of the article it commented on, but
not to the revised version.  Or perhaps a scholium author would like
to drop their content all together (to the extent that this is
possible).  We'll do what we can to accomodate these various actions
in this section.
\end{notate}

\subsection{Editing labels} \label{editing-labels}

\begin{notate}{To edit labels}
If you're going to edit a label, you'd like to do things like insert
new articles, delete old articles, rearrange the order of entries, and
so on.  Preliminary support for this sort of thing is given in this
section.
\end{notate}

\begin{notate}{On `article-menu-insert-new-article'}
This function lets you add new articles to a recently browsed label.

For now, we assume that we really are adding a \emph{new} article
(and not some existing one); and that this new article will
take the quotidian form offered by `make-scholium' (Note 
\ref{make-scholium}).  These assumptions may be relaxed later.
 
The function one of `label-article' and `label-article-insert-before'
to add the label to the new article (Note \ref{label-article}, Note
\ref{label-article-insert-before}).

(When creating this new article, I hope that enough ancillary metadata
, \inp{type} and so on, is added to allow it to be recognized as part
of the relevant hierarchy later.)
\end{notate}

\begin{lisp}
(defun article-menu-insert-new-article ()
  (interactive)
  (if currently-displayed-label
      (let ((new-entry (read-string "New entry: "))
            (old-entry (save-excursion
                         (goto-char (line-beginning-position))
                         (search-forward-regexp "[. >] ." nil t)
                         (setq name-of-current-article
                               (get-text-property (point) 'name)))))
        (setq new-scholium-name new-entry)
        (let ((line (line-number-at-pos)))
          (if old-entry
              (label-article-insert-before
               new-entry old-entry currently-displayed-label)
            (label-article new-entry currently-displayed-label))
          (display-label currently-displayed-label)
          (goto-line line))
        (make-scholium))
    (message "Listing doesn't represent a label.")))

(define-key article-menu-map "i" 'article-menu-insert-new-article)
\end{lisp}

\begin{notate}{Committing edits to labels} \label{commit-label}
So far, the editing steps we do with labels are automatically
entered into the backend, so, at this point, we don't need a 
`commit-label'.  If it turns out that we want one, then these
routines will have to be rewritten somewhat, and `commit-edits'
will have to have `commit-label' hooked into it.
\end{notate}

\subsection{Further notes on editing and deleting}

\begin{notate}{Reparsing and propagating after editing} \label{reparsing-after-editing}
Editing an article is one of the main reasons we'd ever need to do
reparsing.  When text moves around, scholia are likely to be attached
to different regions.  This means we need to change the \inp{about}
elements of attached articles for the collection to stay coherent.
Proper reparsing behavior is necessary for various functions to behave
properly; see for example Note \ref{moving-with-reparsing}.  We
probably will want to add the appropriate reparsing function to the
corresponding after-editing/before-saving-changes hook.
\end{notate}

\begin{notate}{Changing names}
One simple ``edit'' would be changing the name of an article; doing
this of course requires the new name to replace the old name in every
reference to the article.
\end{notate}

\begin{notate}{Changes to an article affect attached articles}
However, changes to the \emph{text} of a given article may affect any
scholia that are \emph{about} that article.  (To say nothing of how
the semantic contents of the change might affect these attached
scholia, the regions may be wrong.)  One approach to take would be to
attempt to propagate the changed buffer positions to all attached
scholia.  Of course, we can't necessarily assume that the owner of the
attached scholia will accept these propagated changes.
\end{notate}

\begin{notate}{We shall adopt a versioned model} \label{we-adopt-a-versioned-model}
These considerations suggest that we should go with the ``versioned''
diachronic model.  According to this model, changes can be propagated,
but on a tentative basis.  If another user doesn't accept your
changes, they will still have a perfectly good scholium attached to a
previous version of your article.
\end{notate}

\begin{notate}{Moving between versions}
It would probably be handy to have this use existing browser code,
e.g. from the local browsing section (Section \ref{local}).
\end{notate}

\begin{notate}{Different ownership models can do different things}
When permissions aren't a concern, we can set the system up so that
propagated changes are automatically accepted.  In addition, we can
set up versioning to resolve at different levels of detail depending
on the application.
\end{notate}

\begin{notate}{Changes outside the system}
One further consideration is the fact that articles may change outside
of the scholium system (e.g. a file may be edited, a remotely-hosted
web page may change, or a process may be killed).  We assume that when
a change like this is noticed, the scholium system will create a new
version of the article \emph{automatically} and propagate appropriate
changes to other documents.
\end{notate}

\begin{notate}{How to do automatic updates}
Noticing changes to files is a bit of a challenge; not insurmountable,
but it does mean that we'll need an internal representation of the
file's contents (i.e. one that we assume is not subject to change in
the same way that the file is!).  As it stands, the system doesn't
include this sort of version management apparatus.  In order to mend
things, whenever an article is created (or edited), a backup copy
should be stored that can serve as a reference.  At least, I guess
this is how version management software works.  Presumably it isn't
hard to add this sort of feature, but before plunging ahead, I think
it would be worth looking into the algorithms and protocols used by
extant version management tools.
\end{notate}

\begin{notate}{When should auto-accept work?}
Although auto-accept is a nice-sounding thing, it may be best to
always have changes approved by a person, since changes to the text
could completely change its meaning -- and threaten to make attached
scholia ridiculous.  Maybe auto-accept is most useful when \emph{one}
person is editing the document, but even then, there may be times when
content changes so much that the attached scholium would become
ridiculous or meaningless.  In this case, maybe auto-accept is good,
but the user should be able to roll back propagated changes easily?

Keeping track of the genesis of a given scholium is important.  Maybe
this procedural info will be marked up itself sometimes?  Certainly
there can be extra user-supplied info, outside of the bookkeeping
info, that describes relationships between documents.
\end{notate}

\begin{notate}{Changes to metadata in a semantically programmed environment}
We assume that changing \inp{type} and \inp{about} data also require
changing the article version.
\end{notate}

\begin{notate}{Using bookkeeping information to track versions}
As if it wasn't obvious, the \inp{bookkeeping} information will be used to
keep track of different article versions.  (Presumably including diffs
between versions or what have you: it would be good to hook the GNU
diff tools into this system.  See also the comment about storing the
``current'' version of articles in full, to check against changes from
the outside.)
\end{notate}

\section{Derivative works} \label{derivative}

\subsection{Introduction to clusions} \label{intro-to-derivatives}

\begin{notate}{Derivative works in the scholium system} \label{derivative-works-in-the-scholium-system}
Section \ref{rendering} focuses on providing useful views \emph{into}
the document.  The matter of assembling \emph{new articles} which
derive from other articles drawn from the document is essentially a
separate problem; we will tackle this problem in this section.  (Other
sorts of derivative or derivative-like works, such as collections and
indices are given their primary treatment elsewhere.)
\end{notate}

\begin{notate}{Different kinds of derivatives} \label{different-kinds-of-derivatives}
\emph{Inclusion} is probably the most straightforward way to assemble
an an article that derives from another article.  Material from a
certain version of the source article is simply inserted into the
current article.  Inclusion is treated in Section \ref{inclusion}.
\emph{Transclusion} is similar to inclusion, except that instead of
simply inserting source text as it is right now, transclusion causes
the source to be tracked as it changes, and its most up-to-date
version to appear automatically in the derivative work when it's
rendered.  Transclusion is treated in Section \ref{transclusion}.
\emph{Identification} is like a two-way version of transclusion:
instead of the source material being inherited by the derivative
article, under identification, the source is shared between several
articles.  Thus, changes in one place will be mirror in each of the
other places.  This is developed in Section \ref{identification}.
\end{notate}

\begin{notate}{Clusions} \label{clusions}
We refer to specific instructions specifying an inclusion,
transclusion, or identification as a \emph{clusion}.  (By analogy to
the morphisms of category theory.)  In this lingo, we could also refer
to identification as \emph{biclusion}.

Both identification and transclusion are represented as links inside
the \inp{text} field of an article, and this property is definitional
for clusions.  

Identification, on the other hand, is currently implemented as a live,
one-time operation, and so it isn't strictly an example.  However,
once we have a proper versioning system working, it would be fairly
easy to do inclusion as a proper clusion like the others (by making it
point at a specific version of a document from which to draw text) --
assuming that is what is desired.  It may be that the user wants to
simply quote from the other document, and doesn't want to do clusion
maintenance.  We'll have to give some thought to what way of operating
is most desirable here.

I haven't thought of any other major classes of clusions (at least,
not yet), but just to point out that any clusion can be composed with
straightforward text operations (e.g. reversing the characters in a
cluded region) to obtain another clusion.

Mainatining clusions is part of a broader problem of maintaining
consistency conditions, see Section \ref{consistency}.
\end{notate}

\begin{notate}{Allowable sources} \label{allowable-sources}
Compare Note \ref{types-of-text}, which talks about the different
allowable kinds of \inp{text}, but note that, one should be able to
clude things other than text.  The set of objects that can be cluded
is same as the set of objects that a scholium can be \inp{about},
i.e., anything a link can point at.
\end{notate}

\begin{notate}{Derivatives and versioning}
In order to get the relevant functions for building derivatives set up
properly, we really should have version management in place.
\end{notate}

\begin{notate}{List processing as a prototypical example} \label{including-transcluding-and-lisp}
The ``{\tt ,@}'' from LISP comes to mind when thinking about
transcluding, whereas shared list structure is like identification.
It would be interesting to spell the relationship out in detail.

One further thought on the subject is that we may sometimes want to
transclude symbols and s-exps of LISP code -- which brings the
discussion full circle pretty quickly.
\end{notate}

\begin{notate}{Boundaryless formulations} \label{boundaryless-formulations}
The simplest cases to deal with appear to be those in which no
boundary is mentioned.  For example, cluding the whole of some article
$A$ into another article $B$.  In practice, sometimes we can stick to
a boundaryless treatment; other times, boundaries are unavoidable.
\end{notate}

\begin{notate}{Clusion algorithms}
It would probably be good to put an overview of things like
\ref{the-transclusion-algorithm} here, so that the subsequent
discussion of markup makes sense.  (We could talk about the easy
work-around for building this document, and contrast it with the more
intensive operations that go on in the *Compilation* buffer (cf. Note
\ref{making-a-compilation}).
\end{notate}

\begin{notate}{Splitting text properties} \label{splitting-text-properties}
Text inserted in the middle of a quotation does not derive from the
same source as the quoted text.  Therefore, it typically should not
take on the `derives-from' markup of the surrounding text.  For a
discussion of inserting within a marked up region in general (i.e.,
not just in the cases having to do with derivatives), see Note
\ref{inserting-text-in-middle-of-marked-up-region}.
\end{notate}

\begin{notate}{Splitting clusions}
Notice that edits in either the source or the recipient 
article can cause a clusion to split, as illustrated in
Figure \ref{Two-ways-to-cause-clusions-to-split}.

\begin{figure}
\begin{center}
\begin{tabular}{c}
\begin{tabular}{ccc}
     $A$      &$\rightarrow$&         $B$   \\
\verb|xxabcd| &             &  \verb|yyabcd| \\
\verb|efgxxx| &             &  \verb|efgyyy| \\
\end{tabular} \\
\ \\

\hline \\

\begin{tabular}{ccc}
     $A'$     &$\rightarrow(\times 2)$&        $B'$   \\
\verb|xxabcd| &                       &  \verb|yyabcd| \\
\verb|xxxefg| &                       &  \verb|efgyyy| \\
\end{tabular} \\

\emph{or} \\
\ \\
\begin{tabular}{ccc}
     $A'$     &$\rightarrow(\times 2)$&        $B'$   \\
\verb|xxabcd| &                       &  \verb|yyabcd| \\
\verb|efgxxx| &                       &  \verb|yyyefg| \\
\end{tabular}
\end{tabular}
\end{center}
\caption{Edits in either $A$ or $B$ can cause a clusion to split
\label{Two-ways-to-cause-clusions-to-split}}
\end{figure}
\end{notate}

\begin{notate}{Masking clusions} \label{masking-clusions}
We want to mask clusions in much the same way we mask links (Note
\ref{masking-links}).  Indeed, clusions \emph{are} essentially links,
by definition (Note \ref{clusions}).
\end{notate}

\begin{notate}{Format of text with derivatives} \label{format-of-text-with-derivatives}
The format of the \inp{text} field for articles with derivatives is as
follows:
\begin{idea}
(twd <string | clusion> ...)
\end{idea}
Each clusion is a link with the corresponding instruction consed
onto the front of it (cf. Note \ref{links}).  The three instructions
are `transclude', `identify', and `include'.

For example,
\begin{idea}
(twd "This Note can be used as part of an example.\n" 
     (transclude "Format of text with derivatives"))
\end{idea}
or,
\begin{idea}
(twd "Part of this Note can be used as part of an example.\n" 
     (transclude "Format of text with derivatives" 
                 (passage 1 25)))
\end{idea}

There is a small break in transparency here; presumably this won't be
a problem (Note \ref{principle-of-transparency}).
\end{notate}

\begin{notate}{Transclusion and freedom} \label{transclusion-and-freedom}
The right to make and circulate \emph{derivative works} is important
(i.e., when it is allowed!).  One easy sort of derivative is a
compilation of all of the sub-objects that have a given property.  For
example, one could put together a collection of all the first
paragraphs drawn from the various sections of an article, or all the
paragraphs that have been marked as ``needs revision'', or
``theoretically interesting'', etc.  Transclusion makes it easy to
build simple, useful, derivatives of this sort, and keep them up to
date as the master document changes.  (Sometimes one would want to use
somewhat special instructions, e.g. in order to collect only the
\emph{new} items matching an otherwise-global search criterion.)
\end{notate}

\begin{notate}{The dependence of the various derivatives on `commit-edits'} \label{derivatives-and-committing}
I think we may have to do a commit after running `transclude-article',
since it involves changing the \inp{text} field.  By contrast, running
`include-article' causes new scholia to be stored automatically, but
we left committing the text field to the user.  So, I'm not sure; if
there is a way to do transclusion without requiring a commit
automatically, that might be kind of nice.

Note also that each of the different derivative styles will have
things to look for whenever they commit \emph{again}.  (E.g. articles
with inclusions should be inspected to see if perhaps all of the
included text has been deleted.)
\end{notate}

\begin{notate}{Derivatives outside of the scholium system}
We leave it up to the user to write a suitable function for quoting
text from scholium system objects to non-scholium system objects (if
the user has any of these!).
\end{notate}

\begin{notate}{Technical difficulties with text properties} \label{difficulties-with-text-properties}
The fundamental difficulty with text properties seems to be that using
them comes at a non-trivial cost.  So we don't use them for everything
(cf. Note \ref{alternative-to-identification}).

When doing transclusion, rather than using a uniform text property, we
need a text property that gives a unique ID to each character within
the marked up region (Figure
\ref{Rearrangement-not-detected-using-uniform-text-properties}, Note
\ref{transclusion-markup-design}).

I've been assuming that \emph{typical} text properties designating the
association of a region with a scholium will be ``sticky'' -- if
someone adds to a marked-up region, the added text will bear the new
markup.  At some point, we might have a non-sticky option.  However,
the text properties associated with transclusions should always be
non-sticky, since text added in the middle of a quoted passage needs
to be treated as an ``island'' in the middle of the transclusion
(which actually gets split in two).

The problem arises in the fact that Emacs makes it hard and/or ugly to
use an individual text property for each transclusion (see Note
\ref{many-scholium-properties}).  The relevant experiment took place
in CVS versions 41 through 43; version 44 was then identical to
version 41.

The point is, that I've been maintaining all of the data about scholia
on precisely one text property, which of course takes on different
values as needed.  However, the ``stickiness'' of each text property
is controlled individually, and I can't continue to use this trick.

I could, I suppose, use one sticky and one non-sticky text property,
but I think that there may be a better approach involving some work on
the text property system itself.  This will need some more
investigation.
\end{notate}

\begin{notate}{Treating clusions} \label{treating-clusions}
The following subsections provide commands for \emph{building clusions
  interactively}, and for \emph{rendering them properly}.
\end{notate}

\subsection{Transclusion} \label{transclusion}

\begin{notate}{On `transclude-article'} \label{transclude-article}
First we check to see whether the article contains
transclusions already, and if not, we put the text field
into the format used for text with transclusions
(see Note \ref{format-of-text-with-derivatives}).

Next, we divide the text of the article in two; everything prior to
point, and everything after point.  (We have to divide the
\emph{rendered} text, fairly clearly; since there might be more than
one different rendering, we hope that the different possible divisions
are consistent, or at least that we know how to do all of the
divisions.)

Of course, we also need to be able to specify the objects that are to
be transcluded!
\end{notate}

\begin{lisp}
;; totally not working yet!
(defun transclude-article (article)
  (interactive (list (read-article-name)))
  (let ((current (get-article name-of-current-article))
        newtext)
    ;; it really doesn't make sense to use something like this here --
    ;; we only deal with the internal format at commit time.
    (unless (and (listp current)
                 (eq (car current) 'twd))
           (setq newtext `(twd
                           ,(scholium-text current))))
    (let ((beg (point)))
      (insert (sch-plain-text (get-article article)))
      ;; this condition isn't really strong enough
      (when name-of-current-article
        (scholium
         `(derives-from ,name-of-current-article ,article ,(genref))
         nil
         `((passage (,name-of-current-article
                     ,beg
                     ,(point))))
         'derives-from
         'system)))))
\end{lisp}

\begin{notate}{Transclusion markup design} \label{transclusion-markup-design}
The idea is that when some object $T$ is transcluded into the current
buffer, $\mathrm{im}(T)$ is marked up in the following way.  Character
$i$ in this image is given a text property that says ``This is
character $i$ of $n$ contiguous characters that have been transcluded
from the object $T$.''  Then, when parsing, if these text properties
show up in new places (e.g. non-contiguously) or if some of the
numbers fail to show up at all, we'll know to initiate new
transclusions or to change the definition of the current one being
scanned.

The reason for doing things this way is similar to the issue discussed
in the ``Not Intervals'' node of the Elisp manual\footnote{(info
  "(elisp)Not Intervals")}.  If the image of a transclusion was simply
marked up with a property that said ``This is the image of the object
$T$ under transclusion,'' then elements of the region that bears this
text property could be rearranged without anything registering at the
system level (Figure
\ref{Rearrangement-not-detected-using-uniform-text-properties}).  This
would pretty much be a disaster.

The idea here is that \emph{we should be able to edit texts coming
  from transclusion} and continue to have the transclusion basically
work.  In Note \ref{splitting-text-properties}, we decided that text
inserted into a quotation should not take on the ``derives-from''
property that the surrounding text bears.  Thus, if a transcluded
region is broken up, or pieces of it are deleted, we assume that new
transclusions are being requested by the user.

These remarks apply to the representation of transclusions in rendered
documents; the internal representation is discussed in Note
\ref{format-of-text-with-derivatives}.

\begin{figure}
\begin{center}
\begin{tabular}{ccc}
\verb|-----#| &             &  \verb|-----#| \\
\verb|----##| &             &  \verb|-#--##| \\
\verb|---###| &$\rightarrow$&  \verb|---###| \\
\verb|--####| &             &  \verb|--##-#| \\
\verb|-#####| &             &  \verb|-#####|
\end{tabular}
\end{center}
\caption{Rearrangement isn't detected using uniform text properties
\label{Rearrangement-not-detected-using-uniform-text-properties}}
\end{figure}
\end{notate}

\begin{notate}{Changing boundaries in source documents} \label{changing-boundaries-in-source-documents}
In Note \ref{transclusion-markup-design}, we talked about adapting
transclusion boundaries to changes in the derivative document.
However, it should be noted that boundaries may shift in source
documents as well: if the object being transcluded is ``the region
between character position 10 and character position 250 from article
$A$'' and a lengthy header is inserted, the idea behind the
transclusion may be completely swept away.  The way to be safest here
is to specify objects without referring to their boundaries, e.g.
``Article $A$'' should be safe.

\emph{There may be a way of maintaining the boundaries recorded on the
  transclusions themselves in such a way that edits to the source
  article are reflected in updated information in the transclusion;
  i.e., if text is inserted into the region that is being transcluded,
  the boundaries could grow, for example.  But doing this seems tricky
  (at least modestly tricker than maintaining backlinks, for example),
  and I think I will hold off on doing anything like this until I get
  some clue that it is actually feasible to do it right.}
\end{notate}

\begin{notate}{The transclusion algorithm} \label{the-transclusion-algorithm}
(1) The first thing is to run a function to select an object
to transclude.

(2) Then we render it to plain text and insert that text in the
current article.  Appropriate text properties indicating that it is a
transclusion are added next.  (If instead of adding a new
transclusion, we simply wish to render an article that includes
transclusions, we must render each transcluded item in this way.)

(3)  We let the user go wild.

(4) If things have changed at commit time, we need to put new
transclusion instructions together.  This new information replaces the
old.  And, in any event, the rendered transclusions are converted to
an internal format for storage (Note \ref{format-of-text-with-derivatives}).
\end{notate}

\begin{notate}{Simple selection makes for expediency} \label{simple-selection-for-transclusion}
In light of the comments in Note
\ref{changing-boundaries-in-source-documents}, for the time being we
choose the simple method of only allowing the transclusion of whole
articles.  There don't appear to be any major theoretical obstructions
to the transclusion of passages, but let's start with something that
will clearly be manageable.  (See Note
\ref{principle-of-minimality}.)
\end{notate}

\begin{notate}{Derivative predicates for transclusions}
Some sort of `derives-from' predicate may have to be maintained for
transclusions, although here such a `derives-from' predicate might be
deduced from the transclusion \inp{text} directly.  (This is similar
the approach we took with inclusions.)
\end{notate}

\subsection{Inclusion} \label{inclusion}

\begin{notate}{On `include-article'} \label{include-article}
It is simple enough to find an article and insert its text at point
(compare `insert-buffer').

%% We then need to add `derives-from' and `derives-to' properties (note
%% similarity to backlinking; indeed, we reuse `put-backlinks', relying
%% on the general way in which it was written; see Note
%% \ref{put-backlinks}).
\end{notate}

\begin{lisp}
(defun include-article (article)
  (interactive (list (read-article-name)))
  (let ((beg (point)))
    (insert (sch-plain-text (get-article article)))
    ;; this condition isn't really strong enough
    (when name-of-current-article
      (scholium
       `(derives-from ,name-of-current-article ,article ,(genref))
       nil
       `((passage (,name-of-current-article
                   ,beg
                   ,(point))))
       'derives-from
       'system))))
\end{lisp}

\begin{notate}{By default, don't print `derives-from' scholia}
Typically we don't need to see `derives-from' scholia, although it
could be handy to write a special display mode in which these scholia
are printed (and probably special semantics for how they are
displayed).
\end{notate}

\begin{lisp}
(add-to-list 'non-printing-types 'derives-from)
\end{lisp}

\begin{notate}{Deleting `derives-from' properties}
If upon reparsing, if no sections are actually marked up with a
`derives-from' property that was previously recorded, it can be
assumed that the text that was quoted from that particular source
article has been deleted.  In which case, odds are that the
formerly-derivative document doesn't derive from that source any
longer.
\end{notate}

\begin{notate}{Add `include-article-formatting-minimally'}
It might be helpful to have a function that would do everything
that the previous version does, but without marking the text up,
or perhaps just running it through `format'.  But I'm going
to hold off on this until the use cases seem clearer to me.
\end{notate}

\begin{notate}{Record version number of included article}
Since inclusions are static, we should probably make note of the
specific version of the source article that the recipient derives
from.
\end{notate}

\begin{notate}{Including passages}
It might be useful to support optional ``beg'' and ``end'' arguments
that would make it possible to quote only a substring of the
\inp{article} that is being included.  This seems most likely to be
useful in a setting in which we select the passage to include
interactively.  (Cf. Note \ref{simple-selection-for-transclusion}.)
\end{notate}

\subsection{Identification} \label{identification}

\begin{notate}{Identification from a design perspective}
Basically, identification is ``like transclusion for editing'': we
have direct write access to the original, as well as read access.
Identification is more powerful than transclusion; and, obviously, one
can only objects that one has permission to modify can be edited
through identification.

Unlike inclusion and transclusion, identified copies should be marked
up with uniform, sticky, text properties.  (The user
should be careful that text from the current buffer that wasn't
intended to be part of the identified copy doesn't get stuck
to the identification by mistake!)
\end{notate}

\begin{notate}{The identification algorithm} \label{the-identification-algorithm}
The object that we're identifying with should be inserted and marked
up with an `identifies-with' text property at render time.  Any
markup of this text (e.g. processing of transclusions or
identifications within the text) should be carried out just as if
the source was being edited directly.  At commit time, text bearing
the `identifies-with' property should be compared with the object
it derives from.  If it has been changed, a new version of that
object should be committed.  As for the derivative article, an
internal representation similar to the one described in Note
\ref{format-of-text-with-derivatives} should be stored.
\end{notate}

\begin{notate}{Precedent for identification} \label{precedent-for-identification}
The emacs package ``all.el'' is abstractly
similar\footnote{{\tt http://ftp.gnu.org/savannah/files/auctex/attic/auctex/all.el}}, as
is the idea of shared list structure (Note
\ref{including-transcluding-and-lisp}).
\end{notate}

\begin{notate}{Identification use case} \label{identification-use-case}
Just for instance, one could use this style of transclusion to render
two articles into the same buffer and cut and paste portions of each
into the other, and have the changes mapped back appropriately.
\end{notate}

\begin{notate}{On `insert-identification-at-point'} \label{insert-identification-at-point}
This is similar to Note \ref{include-article}.  For now,
I'm just going to write it so that you can do identification
with other articles; later, more general objects will be allowed
(Note \ref{allowable-sources}).

We don't actually need to create any scholia,
but we can use the scholia property.  It should
be exactly the same when we render articles that already had
identifications recorded internally.
\end{notate}

\begin{lisp}
(defun insert-identification-at-point (article-name)
  (interactive (list (read-article-name)))
  (let ((beg (point)))
    (insert (sch-plain-text article-name))
    (add-to-scholia-property-within-region
     beg
     (point)
     ;; there aren't any links, so there's no proper linkid
     `(identifies-with ,article-name))))
\end{lisp}

\begin{notate}{On `propagate-changes-to-identification-source'}
This runs within the context of
`store-updates-from-identification-images' (when it runs).
Its purpose is to copy changes from the recipient article
in an identification to the source article.
\end{notate}

\begin{lisp}
(defun propagate-changes-to-identification-source ()
  (when (not (equal (buffer-substring-no-properties
                     (second (car identifications))
                     (third (car identifications)))
                    (sch-plain-text
                      (second (caar identifications)))))
    (let ((source (get-article (second (caar identifications)))))
      ;; compare `commit-edits'.  When we actually do
      ;; document versions, this should update the version
      ;; number and any other metadata about how this
      ;; version was created that we get interested in.
      (scholium (second (caar identifications))
                ;; this is bad, since it is storing
                ;; the rendered text, whereas we should of course
                ;; be storing the formatted-for-internal-storage
                ;; text.  But I suppose this is OK for testing
                ;; purposes (wherein we'll only have one level
                ;; of identification).
                (buffer-substring-no-properties
                 (second (car identifications))
                 (third (car identifications)))
                (scholium-about old-contents)
                (scholium-type old-contents)
                (scholium-bookkeeping old-contents)))))
\end{lisp}

\begin{notate}{On `store-updates-from-identification-images'}
This function will run within the scope of `commit-edits' via the
`commit-edits-hook' (see Note \ref{commit-edits-hook}).
We must first identify the identifications (as it were), which we
do by examining the contents of the `adjusted-markup' list.
Then, once these are found, we form a suitable internal
representation.

Everything \emph{between} identifications should get stored
as plain text, whereas the identifications themselves should
be stored as tags.

In addition to adjusting the internal representation of
the article with the identification commands, we should
adjust the text of the source articles, if necessary.
(For simplicity's sake, we could just store updated text
unconditionally for now.)

Concerning the use of `buffer-substring-no-properties':
zapping the properties is questionable, if
we're planning to commit these various derivative
properties serially.  Will have to think about this later.
I think that since this thing is being used to build
internal representations, zapping text properties
might actually be OK.  But I'm not sure.

For clarification on the use of `adjusted-text', see comments on
`commit-edits'.

Note that the only relevant option besides ``less'' is ``equal''
in the `if' here.
\end{notate}

\begin{lisp}
(defun store-updates-from-identification-images ()
  (let ((identifications (remove-if
                          (lambda (elt)
                            (not (eq (caar elt) 'identifies-with)))
                          adjusted-markup))
        (pt 1)
        formatted-contents)
    (while identifications
      (if (< pt (second (car identifications)))
          (setq formatted-contents
                (append
                 formatted-contents
                 (list (buffer-substring-no-properties
                        pt (second (car identifications)))
                       `(ident ,(second (caar identifications)))))
                pt (third (car identifications)))
        (setq formatted-contents
              (append
               formatted-contents
               (list `(ident ,(second (caar identifications)))))
              pt (third (car identifications))))
;      (propagate-changes-to-identification-source)
      (setq identifications (cdr identifications)))
    (when formatted-contents
      (when (< pt (point-max))
        (setq formatted-contents
              (append
               formatted-contents
               (list (buffer-substring-no-properties
                      pt (point-max))))))
      (setq adjusted-text (list 'twd formatted-contents)))))

(add-hook 'commit-edits-hook 'store-updates-from-identification-images)
\end{lisp}

\begin{notate}{Working identifications into the rendering pathway}
We'll need to modify rendering to look at the ``ident'' objects
in the document's representation, and mark these up as identifications.

The way identifications and transclusions are expanded will be
similar.  (See section \ref{rendering-derivatives} for more on this.)
\end{notate}

\begin{notate}{Visible identification markup}
It would be handy to optionally have overlays showing which regions
are marked up as identifications, and also to have scholia that say
what they are identified to.  (This seems like a good example of a
situation in which it would be be best if the scholia only showed up
when the regions they applied to appeared onscreen.)
\end{notate}

\begin{notate}{Cutting and pasting identifications in multi-user environment}
One should be careful that passages that have been marked up as
identifications, which are copied and pasted into other articles which
don't have the same permission-set associated with them, are no longer
treated as identifications.  Perhaps some identifying information
(`name-of-current-article' being a likely candidate) can be added to
the text property added by `insert-identification-at-point', and some
additional checks put into the commit phase.
\end{notate}

\subsection{Rendering articles containing derivative portions} \label{rendering-derivatives}

\begin{notate}{Cluded parts} \label{cluded-parts}
These variables will be modified by `unwind-derivatives' to contain a
list of the transcluded sections of a rendered document.
Specifically, the items on the list are of the form
\begin{idea}
(<object> <beg> <end>)
\end{idea}
where `object' is a link to the thing being transcluded,
and `beg' and `end' represent the beginning and end positions
of the image of the object under this particular clusion, in
the rendered version of the cluding article.

This information will subsequently be used as part of the markup
routine, where it will allow us to add appropriate text properties
indicating cluded regions.

(It may be advantageous to assume that we can always use a
\emph{cached} version of the article we're cluding.  This could be a
help when the article itself is derivative.)
\end{notate}

\begin{lisp}
(defvar transcluded-parts nil)
(defvar identified-parts nil)
(defvar included-parts nil)
\end{lisp}

\begin{notate}{On `unwind-derivatives'} \label{unwind-derivatives}
This will run within the scope of `sch-plain-text', and make recursive
calls to `sch-plain-text' as needed, to turn cluded texts into strings
suitable for display.   Specifically, a simple test has been added to
`sch-plain-text-hook' that causes `unwind-derivatives' to run whenever
rendering an article whose \inp{text} field is a list that begins with
the token `twd' (see Note
\ref{adding-unwind-derivatives-to-rendering-pathway}).

The boundaries of the cluded regions are recorded for subsequent
use, as described in Note \ref{cluded-parts}.

The contents of the `` *Unwinding Derivatives*'' buffer needs to be
cleared out before this function first runs for it to return the
correct string.  (Um, is this all going to work out properly if we
enter into the fully recursive run of `unwind-derivatives'?)  That is
accomplished by modifications to `scholia-display-pre-update-hook';
see Note \ref{prep-for-rendering-text-with-derivatives}.

The need for (extensive) recursion in this function could be done away
with if we cached a \emph{rendered} version of every article that used
clusions somewhere where it could be found by other articles that
clude from it.

However, if we are going to have things like references from cluded
documents appear in the assembled document, or (in general) if we want
to have access to the scholia attached to the cluded regions (Note
\ref{fancy-derivatives}), we'll either have to pre-render the
derivative components, or come up with some scheme for mapping the
positions of markup associated with these things into suitable
positions in the assembled document.

Note that at present markup will be added by
`add-inclusion-and-transclusion-markup' (Note
\ref{add-inclusion-and-transclusion-markup}).
\end{notate}

\begin{lisp}
(defun unwind-derivatives ()
  (set-buffer (get-buffer-create " *Unwinding Derivatives*"))
  (dolist (elt (cdr text))
    (if (stringp elt)
        (insert elt)
      (let (object)
        (cond
         ((eq (car elt) 'transclude)
          (setq object (cdr elt))
          (setq transcluded-parts (cons `(,object ,(point))
                                        transcluded-parts)))
         ((eq (car elt) 'identify)
          (setq object (cdr elt))
          (setq identified-parts (cons `(,object ,(point))
                                       identified-parts)))
         (t
          (setq object elt)))
        (insert (sch-plain-text object))
        (cond
         ((eq (car elt) 'transclude)
          (setcdr (cdar transcluded-parts) `(,(point))))
         ((eq (car elt) 'ident)
          (setcdr (cdar identified-parts) `(,(point))))))))
  (setq ret (buffer-string)))
\end{lisp}

\begin{notate}{Preparation for rendering text with derivative components} \label{prep-for-rendering-text-with-derivatives}
We need to zap `transcluded-parts', `identified-parts' and the
contents of the buffer in which derivatives are to be unwound before
`unwinding-derivatives' runs.
\end{notate}

\begin{lisp}
(defun prep-for-rendering-text-with-derivative-components ()
  (setq transcluded-parts nil
        identified-parts nil
        included-parts nil)
  (save-excursion
    (set-buffer (get-buffer-create
                 " *Unwinding Derivatives*"))
    (erase-buffer)))

(add-hook 'scholia-display-pre-update-hook
          'prep-for-rendering-text-with-derivative-components)
\end{lisp}

\begin{notate}{Adding `unwind-derivatives' to the rendering pathway} \label{adding-unwind-derivatives-to-rendering-pathway}
We make a call to `unwind-derivatives' in `sch-plain-text' when the
appropriate criterion is satisifed (see Note
\ref{sch-plain-text-hook}).  The relevant criterion is that we are
rendering an object (i.e., something that gives us an article) whose
\inp{text} field is a list that begins with the token `twd'.

(The complex wording here has to do with the matter that we sometimes
render objects that aren't articles, e.g. a link to a certain passage.
We will have to make sure that the criterion used here is correct for
those cases as well as the easy case of rendering a whole article.
See Note \ref{sch-plain-text} for details.)
\end{notate}

\begin{lisp}
(add-hook 'sch-plain-text-hook
          '(lambda ()
             (when (and (listp text) (eq (car text) 'twd))
               (unwind-derivatives))))
\end{lisp}

\begin{notate}{On `add-inclusion-and-transclusion-markup'} \label{add-inclusion-and-transclusion-markup}
For identifications, this should add the same sort of faked-up markup
that we added with `insert-identification-at-point' (Note
\ref{insert-identification-at-point}).

It might be kind of cute to put some ``corners'' in, showing
where the included text begins and ends (see Note \ref{monochrome}).

These modifications to `mark-things-up' appear to be desired
unconditionally (unlike e.g. the modifications to `sch-plain-text' we
just saw in Note
\ref{adding-unwind-derivatives-to-rendering-pathway}).  So we just put
`add-inclusion-and-transclusion-markup' directly on the
`mark-things-up-hook' (cf. Note \ref{mark-things-up-hook}).
\end{notate}

\begin{lisp}
(defun add-inclusion-and-transclusion-markup ()
  (save-excursion
    (set-buffer-to-rendering-target-buffer)
    (dolist (elt identified-parts)
      (add-to-scholia-property-within-region
       (second elt)
       (third elt)
       `(identifies-with ,(first elt))))
    ;; this may end up having to be considerably more complicated
    (dolist (elt transcluded-parts)
      (add-to-scholia-property-within-region
       (second elt)
       (third elt)
       `(transclusion-of ,(first elt))))))

(add-hook 'mark-things-up-hook 'add-inclusion-and-transclusion-markup)
\end{lisp}

\begin{notate}{Should identifications be unwound recursively?}
I think they won't need to be; if someone asks to identify a region
with some other region that also contains identification, then you
might expect that we'd need to follow through to identify the
indicated portion of the new article with that other earlier source.
That's probably the right way to do things; however, I suppose we
could do something simpler in the mean time, since if we don't write
documents that depend on documents with identifications in them, the
case we've been discussing won't come up!  Besides, it would probably
be instructive to have on hand a non-recursive function that does a
similar job to the function written for transclusions.

(Note that labels can sometimes be used to limit complexity of certain
operations, see Note \ref{cost-of-maintaining-labels}.
\end{notate}

\subsection{Expanded and collapsed views} \label{expanded-and-collapsed}

\begin{notate}{Expanded and collapsed views of derivative articles}
There should probably be at least two different views into an article
that makes use of any of the derivative-making tools we've discussed
in this section: one that shows all of the rendered text, and another
which just shows \emph{tokens} standing in for the rendered text.  See
Note \ref{transcluding-and-outlines}, as well as Note
\ref{rendered-and-source-views}.  Collapsed views are essentially
``graphical'' representations of formats like the one listed in
Note \ref{format-of-text-with-derivatives}.  In order to make these views
available to the user, we'll want to provide `expand-derivatives'
and `collapse-derivatives' commands, as well as, probably, individual
commands for individual derivative items and each of the three
derivative types.
\end{notate}

\begin{notate}{View is initially expanded}
New derivative pieces are inserted in their expanded
form (see, e.g., Note \ref{insert-identification-at-point}).
\end{notate}

\begin{notate}{Collapsing and expanding individual derivative items}
We could set things up to render everything as collapsed or
expanded all at once, but another option is to have a function
that collapses or expands individual items.
\end{notate}

\begin{notate}{Identification through nontrivial maps} \label{identification-through-nontrivial-maps}
In some cases it would be useful to have identifications
go through non-trivial maps, for example, to transform
a rendered piece of \LaTeX\ text into the standard source
form.  In addition, it might be nice to store metadata about
Notes in some other place than the note's immediate neighborhood.
\end{notate}

\begin{notate}{Alternative to identification for simple literate programming applications} \label{alternative-to-identification}
It may also be the case that we don't really need to do
identifications to get the ability to import and export
code from the scholium system.   As long as everything
is appearing in Notes, we should be able to edit things
in an outline fashion and then export that to the \LaTeX\
document.
\end{notate}

\subsection{Quick compilations and other listing tricks} \label{quick-compilations}

\begin{notate}{Uses for compilations}
What is presented in this section likely to just be a stop-gap measure
until more general and powerful mechanisms for managing derivatives
are available.  But nevertheless, the basic ideas seem useful.  For
example, when viewing a certain article that is a member of a certain
label, it might be handy to view everything in the label all at once
in a compilation.  And so on and so forth for various compiled views
into a document.
\end{notate}

\begin{notate}{On `find-names-in-listing'}
This can be used to find the names of listed articles.
\end{notate}

\begin{lisp}
(defun find-names-in-listing (just-marked)
  (let ((names (list t)))
    (save-excursion
      (set-buffer (get-buffer-create "*Generic List*"))
      (goto-char (point-min))
      (while (and (not (eobp))
                  (if just-marked
                      (search-forward-regexp "^>" nil t)
                    t))
        (let* ((next-change (next-single-property-change 
                             (point) 'name))
               (prop (when next-change
                       (get-text-property next-change 'name))))
          (if (not next-change)
              (goto-char (point-max))
            (goto-char next-change)
            (when prop 
              (nconc names (list prop)))))))
    (cdr names)))
\end{lisp}

\begin{notate}{Making a compilation from a listing of articles} \label{making-a-compilation}
In Note \ref{bits-and-pieces}, we talk about making all the articles
that match a given criterion into a new article.  A quick way to
get this functionality to the user is to turn an article listing
into a compilation.  

The version here is very preliminary; eventually we'll be putting in
identifications or something like that to make the various pieces of
the compilation ``hot''.

It be nice to have various display options, either features that
screen certain kinds of content in or out at render time
(e.g. printing of attached code snippets could be optionally be made
automatic), or that add actionable features to the display (e.g. to
enable the user to expand and collapse cluded articles, or to select from a cluded label).

We present a couple of variants here.
\end{notate}

\begin{lisp}
(defun listing-to-compilation (just-marked-items)
  (interactive "P")
  (let ((names (find-names-in-listing just-marked-items)))
    (pop-to-buffer (get-buffer-create "*Compilation*"))
    (erase-buffer)
    (dolist (name names)
      (let ((article (get-article name)))
        (insert (upcase 
                 (propertize (format "%s" (scholium-name article))
                             'face 'italic))
                 "\n"
                 (format "%s" (scholium-text article))
                 "\n")))
    (goto-char (point-min))))

(defun listing-to-compilation-1 (just-marked-items)
  (interactive "P")
  (let ((names (find-names-in-listing just-marked-items)))
    (pop-to-buffer (get-buffer-create "*Compilation*"))
    (erase-buffer)
    (dolist (name names)
      (let ((article (get-article name)))
        (insert  "\\begin{notate}{"
                 (propertize (format "%s" (scholium-name article))
                             'face 'italic)
                 "}"
                 "\n"
                 (format "%s" (scholium-text article))
                 "\\end{notate}"
                 "\n\n")
        (dolist (scholium (mapcar (lambda (backlink)
                                    (get-article (car backlink)))
                                  (get-backlinks name)))
          (when (typedata-includes (scholium-type scholium) 'code)
            (insert "\\b" "egin{lisp}\n"
                    (scholium-text scholium)
                    "\\e" "nd{lisp}\n\n")))))
    (goto-char (point-min))))
\end{lisp}

\begin{notate}{Finding code continuations with backlinks} \label{finding-code-continuations}
Establishing the existence of attached code snippets in
`listing-to-compilation-1' (Note \ref{making-a-compilation}) is
currently done in a way that doesn't take full advantage of the
backlink-to-link correspondence we've put together.  This sort of
thing was one reason I thought we might want to use typed backlinks;
but eventually I decided that simply putting backlinks in direct
correspondence with links would be enough, see Note
\ref{implementation-of-backlinks}.
\end{notate}

\begin{notate}{Additional compilation tricks}
Code snippets aren't the only ``bonus features'' we might like to have
appear in a compilation (cf. Note \ref{making-a-compilation}); for
example, we might want to make an article appear together with the
articles that immediately precede and follow it in some linear order.
(This makes the compilation a bit more like browsing a printed
document.)

It would also be handy to have an option for exporting a fully
\LaTeX -ready version of a compilation (compare Note
\ref{selectively-populate-existing-outline}).

Another fun trick that would be fairly cute, when this is all working
quite a bit better, would be to have links that run internally within
the compilation show up in a different color from links that point
outside of the compilation.  (We'll want to develop some code for
making the display jump to just the right point when following an
internal link, or any link that points to a specific portion of a
given article.)  One might want to view one of these intra-linked
articles in place in the compilation, or perhaps instead on its own,
using the normal single-article display.

The sequence of things displayed in the compilation buffer is the sort
of thing that might benefit from having its own history list; see Note
\ref{thematic-histories}.
\end{notate}

\begin{notate}{Selectively populate existing outline} \label{selectively-populate-existing-outline}
It would be kind of interesting to combine `listing-to-compilation'
with `export-scholium-system' (Note \ref{export-scholium-system})
in such a way as to add some particular subset of the
Notes into the section hierarchy.
\end{notate}

\begin{notate}{On `listing-to-label'}
It can be handy to turn an arbitrary listing into
a more permanent label.
\end{notate}

\begin{lisp}
(defun listing-to-label (label-name)
  (interactive "MLabel: ")
  (let ((names (find-names-in-listing)))
    (scholium label-name
              (cdr names)
              nil
              'label
              (sch-book))))
\end{lisp}

\subsection{Further notes on derivative works}

\begin{notate}{Derivatives and copyright}
Recall that for copyright purpose, it is \emph{expression} that is
supposed to be examined when deciding whether a given work derives
from some other work.  This somewhat elusive quality is going to be
harder to capture than the property of being a derivative via direct
quotation and subsequent revision.
\end{notate}

\begin{notate}{Pedigrees} \label{pedigrees}
When adding metadata to show what articles a given article derives
from, we probably only want to bother with the parent, not with
ancestors from earlier generations.  More extensive information about
an article's pedigree can be found by chasing backlinks, as needed.
\end{notate}

\begin{notate}{Aggressive management of derivatives} \label{aggressive-dop}
A more ``aggressive'' solution to recording information about
derivative works would be to mark up rendered articles with a text
property that identifies them; this way, whenever the text is
cut-and-pasted to a new buffer, the text property will be carried
along, and the source of the text can be identified and noted
when the recipient article is reparsed and committed.
\end{notate}

\begin{notate}{Fancy derivatives} \label{fancy-derivatives}
It may be useful at times to carry over certain scholia attached to
the region that is being quoted.  There are probably a number of
variations that people would be interested in trying.
\end{notate}

\begin{notate}{Eventual improved support for virtual collections?} \label{improved-support-for-virtual-collections}
In Note \ref{derivative-works-in-the-scholium-system} we briefly
contrasted derivative articles with collections and indices.  It seems
likely that we can come up with tools that will make it easier to
build useful indices; subcollections (Section \ref{subcollections})
seem to capture the basic idea, but there is room to grow there.
\end{notate}

\begin{notate}{Derivative articles and forking} \label{derivative-articles-and-forking}
One obvious way to fork an article is simple -- to include it in full
and then begin edits there.  The ability to do this gives users quite
a bit of freedom; however, it is worth noting that it can frequently
be to a user's benefit to get their changes inserted in (or around)
the original source, instead of placing them in a separate derivative
version.  (This relates to Section \ref{philosophy}.)
\end{notate}

\begin{notate}{Tracking changes in parents}
Transcluding will make it so that changes to the parent are
immediately reflected in the child.  This is not so with inclusions;
however, owners of articles that use inclusion may still want to be
\emph{updated} when the parent changes.  See Section \ref{bidi}.
\end{notate}

\begin{notate}{Tracking changes in children}
Nevertheless, there are certainly times when it is beneficial or
expedient to create a modified version.  To cater to users in this
situation, we could also set up a derivative-tracking device, who's
function would be to update the original author about the fate and
progress of any derived versions.  See Section \ref{bidi}.
\end{notate}

\begin{notate}{Overviews} \label{overviews}
It would be nice to include markup associating an overview with the
section of the document that was being overviewed.

\emph{Editing (or mixing) down} is essentially similar thing done in a
different spirit.  Whereas I'm imagining that the edited-down version
would replace the old version, the overview would likely exist in
parallel with the original.
\end{notate}

\begin{notate}{Transclusion with modifications}
In addition to the overviews of Note \ref{overviews}, we could be
interested in other sorts of ``transclusion with modifications''.  For
example, if the overview is created by transcluding only the first
paragraph from every chapter, other modified versions could be created
by replacing \emph{foo} by \emph{bar} everywhere.
\end{notate}

\begin{notate}{Transclusion (typically) induces a linear structure} \label{transclusion-induces-linear-structure}
If an article is built out of other transcluded articles, then it
should be possible to abstract an \emph{order}, so that one can say
``next article'' or ``previous article'' and have these things make
sense.  But of course, this order can easily be broken, say by
allowing there to be two ``next articles'' (compare \emph{Literary
  Machines}).  Also note that nested transclusions (typically) induce
a tree structure.
\end{notate}

\begin{notate}{Transclusion-like actionable derivatives}
In order to keep the first example of a scholium given in section
\ref{examples}, ``Remark on Scholium Definition'', up to date, it
would be nice not to have to keep changing the numbers that identify
the the beginning and end of the region being indicated by hand.  This
problem could be solved by putting a scholium in place of each number
that renders as the appropriate number whenever the article is
rendered.  A standard scholium \inp{about} the region in question
would be maintained by reparsing (Section \ref{parsing}), but the
actual numbers that designate the beginning and end of this region
will move around.  Designing the actionable scholium that would
accomplish the job here will take a bit of doing.
\end{notate}

\begin{notate}{Indivisible read-only text for collapsed labels}
Perhaps collapsed text (Note \ref{transcluding-and-outlines}) should
have a special read-only property (alternatively, changes to such text
would constitute suggested changes to the \inp{name} of the article
that was transcluded; see Note \ref{identification-use-case}).
\end{notate}

\begin{notate}{Scholia Display as transclusion}
Although it isn't technically put together that way (at least, not
now), one could think of the Scholia Display as being an article that
is put together by transcluding all of the scholia attached to the
current main article.
\end{notate}

\begin{notate}{Rendering deleted objects} \label{rendering-deleted-objects}
This is a general problem that affects both rendering straightforward
articles which happen to be transcluded and rendering articles with
transclusions that point at articles that have been deleted.
Presumably, some kind of ``404 File not found'' error will be
displayed at an appropriate point in the text.
\end{notate}

\begin{notate}{Building transcluded articles by reparsing}
Particularly if transcluded portions of a document are not directly
editable, it might make sense to build ``text with transclusions'' by
first assembling the content of a buffer containing transclusions, and
then parsing the text properties.  If the transcluded portions were
directly editable, we could get into a somewhat tricky situation
(e.g. should shearing three characters off of the end of the image of
some transcluded text also change the transclusion command so that
those characters are not copied over in the future? -- probably,
although there are probably better ways of editing transclusion
specifics).

This is essentially what I've decided to do; see Note Transclusion
from a design perspective.
\end{notate}

\begin{notate}{Rendering text with derivative pieces to specific places}
Right now, text that includes transclusions is set up to only render
to the ``Main Article Display'' buffer (because the \inp{text} field
is not a buffer).  We could set things up so that articles whose text
contains transclusions could be associated with specific buffers where
the rendered text would appear.  This would require some changes to
`scholia-overwhelm-display' (see Note
\ref{scholia-overwhelm-display}).
\end{notate}

\section{Saving and restoring} \label{saving-and-restoring}

\begin{notate}{Why saving?}
Having created several different scholia, you may wish to save your
work, to read back in later, or to send to a collaborator.
\end{notate}

\begin{notate}{Reverting}
The idea of \emph{reverting} to an earlier revision relates to the
previous section, as well as, vaguely, to the notion of
\emph{restoring} a saved article.  It isn't clear which section it
should go in, since it isn't actually any of these things!  It seems
like a somewhat likely candidate to go into the previous section
instead of this one.
\end{notate}

\begin{notate}{Restoring and managing state}
Various details (e.g. an index counting new articles that have been
added to the system) may need to be maintained between sessions.
\end{notate}

\begin{notate}{Saving buffers to files}
When saving/quitting, articles that come from buffers may then get
associated with files (assuming that the buffers end up getting saved
to a file).  Presumably we should \emph{offer} to save buffers to
files; alternatively, we could save them in some internal format that
the scholium system knows how to restore (reminds me of Emacs
filesets; see also Note \ref{excess-of-read-scholia-file}).
\end{notate}

\begin{notate}{Saving special scholia}
We should presumably offer the option of just saving only \emph{new}
or \emph{modified} scholia, or those that are connected with the
current buffer, or those that have a certain \inp{type}, etc., rather
than saving the whole library.  In some cases, the collection to be
will presumably only be found by predicate matching; other times, we
will be able to maintain an index of the subcollection to be saved
(e.g. we can keep track of all the articles that have been modified
since the last save).
\end{notate}

\begin{notate}{Autosave}
We might consider making some environment variable that would cause
things to be saved to disk automatically.
\end{notate}

\begin{lisp}
(defun save-all-scholia (filename)
  (interactive (list
                (read-file-name "Filename: ")))
  (save-window-excursion
    (gather-scholia)
    (write-file filename)
    (kill-buffer (current-buffer))))
\end{lisp}

\begin{notate}{On `gather-scholia'}
It might be a good idea for this function to take an optional
predicate or label, and gather only scholia that match that predicate.
We could easily cause the gathering function to run only within a
certain context as well.

This function doesn't really have anything to do with saving,
logically speaking, so perhaps it shouldn't go in this section.
\end{notate}

\begin{lisp}
(defun gather-scholia ()
  (interactive)
  (set-buffer (get-buffer-create "*Scholia*"))
  (delete-region (point-min) (point-max))
  (maphash (lambda (name val)
               (write-scholium (cons name val)))
             article-table)
  (display-buffer "*Scholia*"))
\end{lisp}

\begin{notate}{Improving the design of `write-scholium'}
One could probably make these things print a bit nicer, e.g. to make
everything fit within 80 columns, but this seems to be good enough for
the time being.
\end{notate}

\begin{lisp}
(defun write-scholium (article)
  (let ((nl "\n          "))
    (insert
     (concat "(scholium " (maybe-quoted-format (scholium-name article))
             nl (maybe-quoted-format (scholium-text article))
             nl (maybe-quoted-format (scholium-about article))
             nl (format "'%S" (scholium-type article))
             nl (format "'%S" (scholium-bookkeeping article))
             ")\n\n"))))

(defun maybe-quoted-format (obj)
  (if (and (not (null obj))
           (or (atom obj) (consp obj)))
      (format "'%S" obj)
    (format "%S" obj)))
\end{lisp}

\begin{notate}{Design of `read-scholia-file'}
This reads and evaluates all of the scholia that have been written out
into the file stored at \inp{filepath}.
\end{notate}

\begin{lisp}
(defun read-scholia-file (filepath)
  (interactive "fFile: ")
  (find-file-literally filepath)
  (read-scholia-buffer)
  (kill-buffer (current-buffer)))

(defun read-scholia-buffer ()
  (while (condition-case nil
             (eval (read (current-buffer)))
           (error nil))))
\end{lisp}

\begin{notate}{Apparently excessive generality of `read-scholia-file'} \label{excess-of-read-scholia-file}
This system would actually read any elisp file (er, I'm not sure it
would deal well with comments).  There are built-in functions that
accomplish the same thing (`load-file' and `eval-buffer' come to
mind).  The idea here was that we might want to do something somewhat
more complicated than simply evaluating the code found in the file.
That may still happen.
\end{notate}

\begin{lisp}
(defvar search-directory-for-scholia t)

(add-hook 'find-file-hook 'search-directory-for-scholia)

(defun search-directory-for-scholia ()
  (list-directory default-directory t)
  (let ((sch-file
         (replace-regexp-in-string "\\..*" ".sch" (buffer-file-name))))
    (when (search-forward sch-file nil t)
      (read-scholia-file sch-file)
      (display-scholia-about-current-buffer)))
  (kill-buffer "*Directory*"))
\end{lisp}

\section{Distributed authorship} \label{distributed}

\begin{notate}{Introducing distributed authoriship}
Thus far, there has only been one author.  But on a multi-user system,
several different authors would be adding text to the system.  It is
fairly trivial to set up a system-wide list of directories to browse
for articles, enabling users of a shared filesystem to co-author a
digital library together.  We can do something very similar when the
authors are not physically co-located, by using CVS or GNU Arch to
collect the contributions of the various authors.
\end{notate}

\begin{notate}{Masters}
Do we need to have one ``master'' copy of each article in the system,
or can downstream users consider the version that they downloaded to
be the de-facto master copy?  Probably the Arch documentation has
something to say about this.
\end{notate}

\begin{notate}{Differences between the system here and the typical diff-based model}
I'm not completely sure what the advantages of the system here will be
over the typical diff- and email-based system for maintaining code.
One difference is that information will be relayed using a ``pull''
rather than a ``push'' model (see Note \ref{draw}).  Other aspects of
the system may simply be more convenient ways of doing things that
people already do; for example, loading and unloading sets of patches.
(I'm sure there is already plenty of software for doing this
conveniently.)  On this note, compare
filesets\footnote{{\tt http://www.nongnu.org/qualta/FilesetsDoc.html}};
contributed data coming from any given source would likely be loaded
into a special subcollection.
\end{notate}

\begin{notate}{Relationship of namespaces to distributed authorship}
One might consider reading articles from each distinct article source
into its own namespace.  On the other hand, it would be possible
to simply tag the articles with some \inp{bookkeeping} information
to show where they had come from.  The same complexity issues arise
here as in other (possible) applications of the namespace principle.
\end{notate}

\begin{notate}{Merging articles with name conflicts}
From time to time, authors will create scholia about similar things
(e.g. Emacs functions) and someone will want to merge the results.  We
should make it straightforward to do this.  (It may be that the best
way to merge is to just use namespaces and always display the articles
together.)
\end{notate}

\begin{notate}{Scalability of search}
In general, some work will have to be done to keep search spaces
fairly small, but an opt-in model for spaces to search will help with
that (compare RSS).  The algorithms associated with a \emph{global}
version of this system (like Xanadu) would be pretty intensive.  We'll
have to think more about this if a deployed system ever gets big.
\end{notate}

\begin{notate}{Draw} \label{draw}
People should subscribe or not depending on their preferences.  Free
software, if nothing else, teaches us to \emph{look at the demand
  side!}

In distributed implementations, we typically won't \emph{send} updates
as much as make them available for pickup.  We certainly could send
some sort of notification saying that new content is available.  On
the other hand, there's no particular reason \emph{not} to send
content directly in some implementations.  (For more on the topic of
making updates available, see Note
\ref{send-messages-to-redirected-scholia}.)
\end{notate}

\begin{notate}{FAIF not FAIB, and DRM}
To think about this another way: the Xanadu system is FAIF but not
FAIB.  The idea was to charge people on a pay-per-view (or
pay-per-use) basis, with some fraction of royalties going back to each
of the authors whose work made up part of the delivered text.  (It
would be interesting to try to connect -- or to dissociate -- this
model from current DRM ideas.)
\end{notate}

\begin{notate}{The p2p connection}
Assembling a document from various disparate sources in the database,
reminds me of something that comes up in p2p filesharing, namely the
way in which documents can be assembled from partial data drawn from
various nodes.  How should one go about the process of assembling the
document from the sources available?  We will need to think and learn
more about the algorithms that would need to be employed in a system
like the one described here, including its potential uses in
distributed content systems.
\end{notate}

\begin{notate}{Increasingly expensive local operations}
Assembling massive amounts of information, e.g., sorting the scholia
about a given document, is going to be somewhat computationally
intense ($n \log n$, right?).  If we ever got into situations in which
there were thousands of scholia about any given article, we might want
to avoid doing sorting (perhaps by storing the scholia in sorted order
some place or other).  Also, by default, we should avoid sorting any
scholia we don't need to display (e.g. frequently, actionable
scholia), because sorting them is just a waste of time.
\end{notate}

\begin{notate}{Reasons things might not bubble up too bad}
My \emph{guess} is that even a very popular article won't have a whole
lot of printing scholia attached to it.  A number of people will want
to make comments, sure, but others will prefer to make derived
versions, or comments on the 1st generation of comments.
\end{notate}

\begin{notate}{Are names always available?}
I'm not sure whether an article would ever appear that had been marked
up with names or namespaces that we don't have access to.  Maybe.
This seems more likely to be an issue in
real life when we're dealing with distributed documents, but it could
potentially be an issue whenever there are multiple namespaces
(or even one poorly maintained namespace; compare Note \ref{rendering-deleted-objects} --
sometimes articles will be deleted).
\end{notate}

\begin{notate}{Accepting updated `about' information} \label{accepting-updated-about-info}
Why would an author want to incorporate updated \inp{about}
information into the original article?  One reason is simply to check
that this information is accurate.  By allowing users to set updated
\inp{about} data automatically when editing a scholium's target, a
degree of flexibility is built into the system that would be
impossible (and, indeed, probably wouldn't make sense) otherwise.
But when one's work is being interpreted in a new way, and, indeed,
in a new context, it may be worthwhile to check that you agree with
this interpretation.
\end{notate}

\subsection{Bidirectional updates} \label{bidi}

\begin{notate}{Setup for bidirectional updating}
We'll assume that one of the authors is using the scholium system, and
the other isn't.  (The other possible cases have either been covered
or are not part of our mandate.)

In order to make bidirectional updating work when some of the editing
is going on outside of the scholium-based system, we would need to be
able to read a stream of diffs coming from the other system and decide
how to incorporate the modifications.  And we also need to be able to
provide them with a stream of diffs that they can use however they see
fit.  We'll certainly have to think more about how to do this!
\end{notate}

\begin{notate}{Using simulation to do bidirectional updating} \label{simulation-for-bidi}
It seems to me that the most useful approach for someone working
within the scholium system and trying to do bidi updates with a
non-scholium system would be to run a simulated version of the
non-scholium system inside the scholium system.
\end{notate}

\begin{notate}{The simplest case}
The simplest case of bidirectional updating is \emph{identification},
which is treated in Section \ref{identification}.  Even this case has
some complexity to it in the distributed case; updates coming from
multiple authors presumably need to be formatted into branching
derivative versions.  (Actually, this is an $n$-directional updating
problem; Arch has had some things to say about this, but we can
probably say more.)
\end{notate}

\subsection{Interoperability} \label{interoperability}

\begin{notate}{Exporting} \label{exporting}
It should be possible to reduce metadata and so on to some format that
would work with external programs (e.g. database programs).  I'm not
at all sure how the use of text properties would be replaced in other
programs, but I think that, in theory, the things we save (Section
\ref{saving-and-restoring}) should be sufficient and readable for any
other interested program.  I'd be interested to know the thoughts of
others on building other interfaces to the scholium system.
\end{notate}

\subsection{A simple scheme for multiuser cooperation} \label{multiuser-cooperation}

\begin{notate}{Crosscut by users and dates}
A simple scheme for facilitating interactions between several users is
as follows.  Each user gets a directory to call their own.  Each users
scholia for the day are stored in a file in this directory.  To catch
up with the activities of all users, one simply loads all of the files
that have appeared since the last load date.

To implement this scheme, we need a method for populating the
data-store and another for scanning it to determine which files to
load.  

We'll assume that only the scholia that bear a certain label (say ``to
publish'') will be published, and that once they have been published,
this label is wiped out (maybe it is set to ``last published'').  In
general, the ``to publish'' set could be determined by some somewhat
complicated algorithm, not just by looking at a label.  

Also, we should presumably maintain all of the imported scholia in a
namespace (most likely, certainly some subcollection) in order to keep
them straight.  (For a first draft, we could just import everything to
the main article table, given that namespaces don't completely work
yet.)

Presumably we should start by building the infrastructure in our CVS
directory.  (Some more general formulation that allows collaboration
between different people writing to disparate locations online would
be good to have eventually; to support this we'll want some sort of
protocol, maybe something like RSS.)
\end{notate}

\begin{notate}{Networked blotters}
Keeping one another up to date on the important features of daily
goings on is an important use of the multiuser system.  The notion
that each user would maintain some sort of ``blotter'', possibly
incorporating some auto-generated content, which others can read to
stay in the loop.
\end{notate}

\begin{notate}{Strategic usefulness of building and using collaboration tools early on} \label{collaborate-early}
While we could ask people to work on other things first (and to some
extent it will be necessary for people to work on other things all
along the way) it seems likely that using powerful collaboration tools
will help people get things done on other projects or subprojects.
When many people can share differing views on a topic, the discussions
of this topic will develop a richness that wouldn't be so likely to
develop otherwise.  A system that can take into account many views in
a coherent and useful way is going to be a powerful system.  (It would
be good to do some research on the differing abilities of various
extant systems to accomplish this.)

Collaboration is a challenging thing to deal with.  You have greenhorn
or yahoo types who want to charge off and do things on their own (I'm
often one of them), and in the mean time, you don't always have a
central point for collaboration on some topic where you might hope one
would be.  It isn't always clear when you have a certain vision
whether there are others out there who would share that vision or not.
Partly, that's because when you have a vision it isn't always
completely clear what that vision is; but these things interact.
\end{notate}

\section{Bindings and environment variables} \label{bindings}

\begin{notate}{Room to improve on bindings}
This is just an attempt to make something usable, but it is probably
nowhere near close to optimal.  One thing that would be nice would be
to have some prefix for browsing relative to all scholia versus
some other prefix for browsing relative to only the current scholium.
Thus, you could cycle between regions for the current scholium or
regions for all scholia (or go to the first region or whatever)
depending on what prefix you specified -- but the final keystroke
(``f'' for forward or whatever) would be the same.  Perhaps we
could substitute in the first register (as opposed to the second)
for changing the general style of browsing (e.g. from scholia to
temporal or whatever), but keep second and third pretty much the
same.  It would be good to spell out the analogies in a somewhat
detailed table.
\end{notate}

\begin{lisp}
(mapc
 (lambda (elt) (global-set-key (eval (car elt)) (eval (cdr elt))))
 '(([?\C-\;] . (make-keymap))
   ([?\C-\;?m] . (make-sparse-keymap))
   ([?\C-\;?d] . (make-sparse-keymap))
   ([?\C-\;?o] . (make-sparse-keymap))
   ([?\C-\;?s] . (make-sparse-keymap))

   ((kbd "C-; c") . 'commit-edits)
   ((kbd "C-; g") . 'gather-scholia)
   ((kbd "C-; f") . 'follow-reference-or-scholium)
   ((kbd "C-; n") . 'name-of-current-scholium)
   ((kbd "C-; r") . 'save-all-scholia)

   ((kbd "C-; v n") . 'back-to-normal)
   ((kbd "C-; v o") . 'back-to-other-view)

   ((kbd "C-; b b") . 'sb-back)
   ((kbd "C-; b f") . 'sb-forward)

   ((kbd "C-; l a") . 'article-menu-list-all-articles)
   ((kbd "C-; l l") . 'article-menu-list-labels)
   ((kbd "C-; l m") . 'article-menu-list-articles-matching-regexp)
   ((kbd "C-; l d") . 'article-menu-list-metadata-articles)
   ((kbd "C-; l p") . 'article-menu-list-plain-articles)

   ((kbd "C-; m a") . 'make-scholium-about-current-article)
   ((kbd "C-; m b") . 'make-scholium-about-current-buffer)
   ((kbd "C-; m l") . 'make-scholium-about-current-line)
   ((kbd "C-; m n") . 'make-new-undirected-article)
   ((kbd "C-; m i") . 'make-current-buffer-into-article)
   ((kbd "C-; m p") . 'make-scholium-about-part-of-current-article)
   ((kbd "C-; m P") .
    'make-scholium-about-several-parts-of-current-article)
   ((kbd "C-; m s") . 'make-scholium-about-current-scholium)

   ((kbd "C-; d a") . 'display-article)
   ((kbd "C-; d b") . 'display-scholia-about-current-buffer)
   ((kbd "C-; d l") . 'display-article-listing)
   ((kbd "C-; d p") .
    'display-an-article-that-current-article-is-about)
   ((kbd "C-; d c") .
    'display-an-article-that-current-scholium-is-about)
   ((kbd "C-; d r") . 'redisplay-article)

   ((kbd "C-; o y") . 'sch-turn-main-article-overlays-on)
   ((kbd "C-; o n") . 'sch-turn-main-article-overlays-off)

   ((kbd "C-; s n") .
    'scroll-article-display-to-next-region-for-current-scholium)
   ((kbd "C-; s p") .
    'scroll-article-display-to-previous-region-for-current-scholium)
   ((kbd "C-; s b") . 'move-to-previous-region-with-scholium)
   ((kbd "C-; s f") . 'move-to-next-region-with-scholium)
   ((quote [S-tab]) . 'move-to-next-region-with-scholium)
   ((kbd "C-; s a") . 'move-to-first-region-for-current-scholium)))
\end{lisp}

\begin{notate}{Style of code for bindings}
I've heard that using `eval' is typically bad style,
but something had to be done to make these bindings work.
What would have been better in terms of coding style?
\end{notate}

\begin{notate}{Environment variables} \label{environment-variables} 
It would be handy to be able to have things set up so that e.g. every
buffer is automatically an article, too.  Various innovations along
these lines should be added here.
\end{notate}

\begin{notate}{Automatically loading scholia}
Along the lines of Note \ref{environment-variables}, this section
would be a good place to put a variable that would cause scholia to be
loaded for all buffers (which I think says slightly more, or anyway
something slightly different from, the statement that every buffer
would automatically \emph{be} an article).  Note that whenever loading
scholia for a given buffer, there may be several different places to
look to find those scholia: in the article table, but also in local
``.sch'' files, or other places.
\end{notate}

\section{Experiments} \label{experiments}

\begin{notate}{Experiments to try}
This section should describe our experiments modeling different CBPP
systems that are out there in the world -- and coming up with new ones
that we like better.

In this section, we can describe some separate packages that provide
more interesting or advanced display features, for example.  Also
maybe some packages that do things with code (e.g. interactive
text-based games).  Wiki portage would be one basic thing to try.
Finally, of course, it would presumably be useful to get the HDM
codebase into the scholium system.

\begin{itemize}
\item Importing a wiki and getting the scholium system set up
to do wiki-like things.
\item Using the scholium system to write a play, novel, screenplay, or
  an interactive fiction type of game (actionable scholia would
  probably be part of just about any game).  Hipster the
  Gathering\footnote{{\tt
      http://planetx.cc.vt.edu/AsteroidMeta/Hipster\_the\_Gathering}}
  would presumably be an easy game to code up.  My forays into ``fine
  literature'' with the Penmaster's Apprentice saga could also be
  typed up.  These things are currently interesting examples of
  physical scholia (see Section \ref{more-physical-scholia}).
\item Using the scholium system to maintain a text-based forum
 or set of fora, as found on PM or slashdot.
\item Using the scholium system to manage an evolving codebase,
  e.g. HDM content, say APM and APM-$\Xi$, or this document itself
  (making the system interoperable with \LaTeX\ would be important).
\item Using the scholium system to manage TODO lists (like we used to
  do with TODL etc.).
\item Port GCIDE to Emacs and give instant access to definitions from
  within the browser.  Make GCIDE scholiumific (supplying semantically
  close definitions or whatever?).
\item do something like mmm-mode but using scholia about the buffer
  instead of begin/end regexps?
\item Make a semantic network describing relationships between words
  and use it for writing poetry.
\item make a system for exporting to other formats (see Note
  \ref{rendered-and-source-views})
\item Write text-generation software that will provide different ways
  of viewing the network's structure.
\end{itemize}
\end{notate}

\begin{notate}{Cross-referencing in \LaTeX} \label{cross-refs-in-latex}
Note that the system of labels and references in \LaTeX\ allow you to
make ``hypertextual'' linear documents.  This system could perhaps be
extended (maybe by exploiting the scholium system).  It would be cool
to be able to cross-reference the relevant label in some other
document and have a suitable reference show up here.  This is a
scholium-like thing: how to make it work in \LaTeX?  (Maybe
\verb|\pmxref| should be able to refer to anything that has a
\verb|\label| about it.  Perhaps this would work already.)  This also
relates to the topic of Namespaces, see Section \ref{subcollections}.
\end{notate}

\begin{notate}{A comprehensive regression suite}
This would be a fairly simple experiment, which would nevertheless
be very helpful for programming the system.  Namely, create 
a short program that tries to do everything that you can do with
the system.
\end{notate}

\begin{notate}{Go moves}
It would be nice to have a browsable database of Go moves and games
with user annotations: presumably the already-existing Go resources
online include things like this.  If the scholium system can help
improve upon these resources, that would be cool.  For example,
perhaps gnugo's move book could be simultaneously an instruction
manual for human users.
\end{notate}

\begin{notate}{Games built of scholia}
Note that the relationships of people to rooms (containment), people
to their backpacks (carrying), and so forth -- the basic elements of
text-based games -- are all interpretable as scholia-based.  The
``fantasy'' tie-in of these sorts of games works reasonably well with
the name Arxana.  If we're going to push development along this route,
it would be good to do some reading on MUD-like systems.
\end{notate}

\begin{notate}{Stress testing the system}
It would probably be worth doing some empirical tests to see what
happens in a library full of a bunch of junk.  How long does it take
to find and display the one relevant scholium?  How much are things
helped by storing articles in namespaces?
\end{notate}

\begin{notate}{Scholia about functions}
This function, or something like it, seems especially important if
we're going to use the scholium system to manage code.  Every time we
create a function, for example, it would be good to create a scholium
attached to that function (perhaps a scholium object whose \inp{name}
is the function's symbol name, though there could be the need to
disambiguate between the function and variable versions of the same
symbol) with a ``defined-in-buffer'' link \inp{about} the buffer in
which the given function definition is found; Emacs already has
functionality like this in its help subsystem.  This could be done by
explicit command or just on-the-fly by some appropriately-constructed
automatic function.

In addition, one could create scholia that say, for instance, which
functions that function calls.  Then, later, one could assemble a
who-calls or who-is-called listing about a given function, for the
user to browse.  Although this can be done easily enough already
(there is a who-calls.el), it is my sense that this approach would
save on search.  Browsing code at the ``who calls'' level seems
potentially interesting; it would be worth investigating the prior art
here more.
\end{notate}

\begin{notate}{Barkov's ``Hamlet''} \label{barkovs-hamlet}
The system could be used to handily create different versions of a
play, e.g., to write a version of ``Hamlet'' manifesting the view that
actually Horatio is the
prince\footnote{{\tt http://www.geocities.com/shakesp\_marlowe/index.html}} or to
splice in ``Rosencrantz and Guildenstern are Dead'' by Stoppard or
``King of Shreds and Patches'' by Elrord or whatever (assuming that
doing so was legally copacetic; for personal use of course it would
be).  Similarly, our treatment could describe theories about how the
play relates to things outside the play, or (being even more meta),
how the \emph{interpretation} of the play relates to the presentation,
or (still more), where the interpretation comes from, and how I think
\emph{my} interpretation of these various things gets involved.
\end{notate}

\begin{notate}{Using scholia on PlanetMath}
Since articles in the encyclopedia are supposed to describe known
math, they aren't really the right place to describe new insights,
even if these new insights occur while reading that article.  (In
theory such insights could be discussed in the attached forum along
with basic questions or whatever; but a catch-all forum or collection
of catch-all fora and postings without metadata may not be the most
conducive to research-oriented work.)  My thought is, why not have an
attached discussion not in the encyclopedia for talking about some
particular advanced question.  This area can itself use encyclopedia
definitions as scholia to help orient casual readers.  This way,
treatments of established and ``live'' mathematics can develop
together.
\end{notate}

\begin{notate}{Implement a web browser that works with the scholium system} \label{nero2}
Maybe I'll write this before long?  It seems like the sort of thing
that could be very helpful; search through scholia on web pages you've
browsed (and also download everything you browse and save it to
disk, so that you can search through the web pages themselves too).

Through this sort of approach, you might imagine the web (or portions
thereof) being  ``embedded'' in a more advanced hypertext system, at
least for some users.  Something similar can be done for working with
Project Gutenberg\footnote{{\tt http://www.promo.net/pg}} or any
collection of computer-readable text.
\end{notate}

\begin{notate}{Use the scholium system to assist in literature review} \label{literature-review}
There are a number of different literatures that are relevant to the
scholium system.  From artificial intelligence, we have things like KM
or SNePS that are available as free software, and which can
accordingly be reviewed \emph{in their entirety}.  Or, at least in as
much of their entirety as seems useful.  It may be desirable to
simulate or at least partially simulate these systems (which is partly
what the \emph{preliminary} literature review is supposed to
determine, by looking at the basic theory of these systems).

In hypertext, there are systems like No\"osphere which can be treated
similarly.  Indeed, I am planning a meetup with No\"osphere's author,
Aaron Krowne, for purposes of mutual exchange over and potential
cross-breeding between these two systems.

There are no doubt many other interesting literature things to study
out there.  Deciding which ones will be most profitable to look at is
an interesting challenge.
\end{notate}

\begin{notate}{Backlink tricks} \label{backlink-tricks}
One obvious thing to do with backlinks is to compute the
most-linked-to article, or, even better, to rank articles by their
popularity according to in-links.  This can be done by mapping a very
simple function over the \inp{metadata} label (or the ranking could be
maintained on the fly).
\end{notate}

\subsection{Some extremely simple example to show people what's going on}

\begin{notate}{Simplest example}
It would be handy to create a very simple example of a buffer
together with scholia.
\end{notate}

\begin{notate}{Next-simplest example}
After that, it would be handy to make an example of a properly
scholia-based document (showing some richer graph structure).
\end{notate}

\begin{notate}{Various other simple examples?}
And while we're at it, maybe later a simple example of transclusion or
identification or whatever.
\end{notate}

\subsection{Use the scholium system to maintain a library of projects} \label{library}

\begin{notate}{Managing several projects}
We'll do several different experiments, some of which will involve
corpora that are (at least essentially) distinct.  One high-level
object to point at the ``main pages'' of these various projects might
come in handy.  There could be a number of other more advanced
library-like features to add later.
\end{notate}

\begin{notate}{Managing workflow}
In addition to simply presenting output of these projects, it would be
helpful to have the system set up to manage workflow (tasks and so
on).  For now, see \ref{appendix-cal}.
\end{notate}

\begin{notate}{Library of major works}
We have an overall structure for the digital library as a whole worked
out already (Section \ref{digital-library}), but not yet a specific
list of major works.  It is convenient to have an index of these (just
like, later, it will be convenient to have many different indices of
different kinds of documents with different features).  So, we create
a virtual library to index these works in.  (Once this is set up
properly, of course the imported version of the scholium system itself
will be included.)
\end{notate}

\begin{lisp}
(scholium 'major-articles nil nil 'label 'system)
\end{lisp}

\subsection{Use the scholium system to do literate programming} \label{literate}

\begin{notate}{Importing LaTeX docs} \label{importing-latex-docs}
Note that importing a \LaTeX\ document (this one, in particular)
wouldn't be so different from importing a wiki (see Section
\ref{wiki}).

One of the issues is how we're going to represent cross references
(see Note \ref{cross-refs-in-latex}).  In compiled \LaTeX, they
typically appear as a number, whereas in source they appear as a tag.
In the scholium system, reference markup should presumably be used,
and I also suppose that we may as well use the name of the article
being referred to directly.  So, for example, this article would
be linked to by text that read ``see `Importing LaTeX docs'.''
This is similar to the way references appear in Texinfo.

Outside references (footnotes) could be rendered in a different
color (Note \ref{fancy-references}), and made browsable within
the scholium system (Note \ref{nero2}).

On the implementation: `end' might be followed by a lisp expression
that should be attached to the note, but this should be fun for a
trial run.

The format of the scholia representing the notes is kind of weird.  We
shouldn't have junk in the text field of the article.  It would
probably make more sense for items of type "note" to be rendered
specially (if one wished) and maybe to store the tag as a part of a
formatted text field, or, more likely, part of the \emph{type} data.

I think I'd be relatively comfortable adding identification
properties to the regions of the larger document, to get
them to inherit from the individual pieces.  Later we could
parse the section structure.

It would, generally speaking, be a good idea if lower levels
in the hierarchy were \inp{about} their parents in such a way
that we could easily move ``up''.
\end{notate}

\begin{lisp}
(add-hook 'scholia-display-post-update-hook 'text-mode)

(add-to-list 'modified-type-labels '(note . note) t)
(add-to-list 'modified-type-labels '(section . section) t)
(add-to-list 'modified-type-labels '(subsection . subsection) t)
(add-to-list 'modified-type-labels '(subsubsection . subsubsection) t)
\end{lisp}

\begin{notate}{On `map-label'} \label{map-label}
This gives you a way to apply a \inp{function} to every article that
bears a given \inp{label}.
\end{notate}

\begin{lisp}
(defun map-label (function label)
  (mapc function (scholium-text (get-article label))))
\end{lisp}

\begin{notate}{Example of mapping across a label} \label{mapping-across-a-subcollection}
Notice that we use `map-label' (Note \ref{map-label}) in the APM-$\Xi$
importing routine, to convert once-processed content to a more useful
format.  See Note \ref{chunk-out-apmxi-definitions}.
\end{notate}

\begin{notate}{On `swap-out-latex-references'}
Does the opposite of `swap-in-latex-references' (Note 
\ref{swap-in-latex-references}): that is, this
function makes \verb|\ref| tags
in the text look like hyperlinks.
\end{notate}

\begin{lisp}
(defun swap-out-latex-references ()
  (let ((tags-alist (scholium-text (get-article
                                    'Tag-to-Name-converter))))
    (map-label (lambda (name)
                 (with-temp-buffer
                   (let ((article (get-article name)))
                     (insert (scholium-text article))
                     (goto-char (point-min))
                     (while (re-search-forward
                             ;; I don't think we use "xrefs" any more.
                             "\\\\\\(x\\)?ref{\\([^}]+\\)}" nil t)
                       (let ((target (cdr
                                      (assoc (match-string 2)
                                             tags-alist))))
                         (when target
                           (replace-match target t t)
                           (let ((name-of-current-article name))
                             (make-reference-in-current-article
                              (match-beginning 0)
                              (+ (match-beginning 0) (length target))
                              target)))))
                     ;; we don't want these references to be fake
                     (scholium name
                               (buffer-substring-no-properties
                                (point-min)
                                (point-max))
                               (scholium-about article)
                               '(note)))))
               'note)))
\end{lisp}

\begin{notate}{On `import-scholium-system'} \label{import-scholium-system}
For now, this is just `import-sections', but it would be good if we
could do something about sections that contain neither subsections nor
notes, for example, the preface.  Other nice features (e.g. creating
``function type'' scholia for functions, as in Note
\ref{elisp-function-type}) should be added too, eventually.  Sometimes
work arounds can be applied for the time being.

\emph{Hint: I've found that for now one has to `(clrhash
  article-table)' before running this function for things to work
  properly.  I intend to address this issue soon.}

(We might want to make this article use some kind of identification
routine; alternatively, make it work with a ``large structure'' map of
the document's contents.)
\end{notate}

\begin{lisp}
(defun import-scholium-system ()
  (interactive)
  (import-sections)
;  (label-article 'section 'major-articles)
  (swap-out-latex-references))
\end{lisp}

\begin{notate}{Shorthand for editing the text of a scholium}
It would certainly be nice to have some short-hand
from the "editing" section to make it so that we didn't
have to use the `scholium' function every time we wanted
to edit some article.
\end{notate}

\begin{notate}{Autobacklink to label} \label{autobacklink-to-label}
Also, it would be nice if the mere inclusion of ``section'' in the
``major-articles'' label meant that there would be a ``parent''
(back)link from ``section'' to ``major-articles'' (see Note
\ref{import-scholium-system} for the source of this example).

Such backlinks would let you navigate from things that bear a certain
label to the label itself and thence to the peers within that label.
(This certainly seems like a good idea!)
\end{notate}

\begin{notate}{On `import-sections'}
For each section, import any notes and any subsections.

I'm going to try to get this to import front- and back-matter (before
the first section and after the last note, respectively).  This will
make it easier to typeset everything with one command.
\end{notate}

\begin{lisp}
(defun import-sections ()
  (save-excursion
    (set-buffer "sbdm4cbpp.tex")
    (goto-char (point-min))
    (search-forward-regexp "^\\\\section{Prelude}")
    (goto-char (match-beginning 0))
    (scholium "Scholium system frontmatter"
              (buffer-substring-no-properties (point-min) (point))
              nil
              '(note))
    (while (re-search-forward
            (concat
             "^\\\\section\\*?{\\([^}\n]*\\)}"
             "\\( +\\\\label{\\)?"
             "\\([^}\n]*\\)?")
            nil t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (section-end (save-excursion
                            (search-forward-regexp
                             "^\\\\section{.*" nil t)))
             (notes-end (or (save-excursion
                              (search-forward-regexp
                               "^\\\\subsection{.*"
                               section-end t))
                            section-end))
             (notes (let ((current-parent name))
                      (import-notes)))
             (subsections (let ((current-parent name))
                            (import-subsections))))
        (when (not (equal tag ""))
          (scholium 'Tag-to-Name-converter
                    (add-to-or-start-list
                     (scholium-text (get-article
                                     'Tag-to-Name-converter))
                     `(,tag . ,name))))
        (scholium name
                  `(,@notes
                    ,@subsections)
                  '((section parent))
                  '(section label))))
    (goto-char (point-max))
    (search-backward-regexp "^\\\\clearpage")
    (scholium "Scholium system backmatter"
              (buffer-substring-no-properties (point) (point-max))
              nil
              '(note))))
\end{lisp}

\begin{notate}{On `import-subsections'}
For each subsection, import any notes and any subsubsections.
notes are imported if they appear before the first subsubsection,
if there is one, or before the end of the subsection, otherwise,
or before the end of the section, if this is the last subsection
\end{notate}

\begin{lisp}
(defun import-subsections ()
  (let (subsections)
    (while (re-search-forward
            (concat
             "^\\\\subsection{\\([^}\n]*\\)}"
             "\\( +\\\\label{\\)?"
             "\\([^}\n]*\\)?")
            section-end t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (subsection-end (or (save-excursion
                                   (search-forward-regexp
                                    "^\\\\subsection{.*"
                                    section-end t))
                                 section-end))
             (notes-end (or (save-excursion
                              (search-forward-regexp
                               "^\\\\subsubsection{.*"
                               subsection-end t))
                            subsection-end
                            section-end))
             (notes (let ((current-parent name))
                      (import-notes)))
             (subsubsections (let ((current-parent name))
                               (import-subsubsections))))
        (when (not (equal tag ""))
          (scholium 'Tag-to-Name-converter
                    (add-to-or-start-list
                     (scholium-text (get-article
                                     'Tag-to-Name-converter))
                     `(,tag . ,name))))
        (scholium name
                  `(,@notes
                    ,@subsubsections)
                  `((,current-parent parent))
                  '(subsection label))
        (setq subsections
              (append subsections (list name)))))
    subsections))
\end{lisp}

\begin{notate}{On `import-subsubsections'}
For each subsubsection, import any notes that appear before
the end of the subsubsection (if this is the last subsection,
then before the end of the subsection, etc.).
\end{notate}

\begin{lisp}
(defun import-subsubsections ()
  (let (subsubsections)
    (while (re-search-forward
            (concat
             "^\\\\subsubsection{\\([^}\n]*\\)}"
             "\\( +\\\\label{\\)?"
             "\\([^}\n]*\\)?")
            subsection-end t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (notes-end (or (save-excursion
                              (search-forward-regexp
                               "^\\\\subsubsection{.*"
                               subsection-end t))
                            subsection-end
                            section-end))
             (notes (let ((current-parent name))
                      (import-notes))))
        (when (not (equal tag ""))
          (scholium 'Tag-to-Name-converter
                    (add-to-or-start-list
                     (scholium-text (get-article
                                     'Tag-to-Name-converter))
                     `(,tag . ,name))))
        (scholium name
                  notes
                  `((,current-parent parent))
                  `(subsubsection label))
        (setq subsubsections
              (append subsubsections (list name)))))
    subsubsections))
\end{lisp}

\begin{notate}{On `import-notes'}
This imports the notes at the current level in the
section hierarchy.
\end{notate}

\begin{lisp}
(defun import-notes ()
  (let (notes)
    (while
        (re-search-forward (concat "\\\\begin{notate}"
                                   "{\\([^}\n]*\\)}"
                                   "\\( +\\\\label{\\)?"
                                   "\\([^}\n]*\\)?")
                           notes-end t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (beg (progn (next-line 1)
                         (line-beginning-position)))
             ;; no need to bound the search for the end, because we
             ;; assume that every "notate" environment is actually
             ;; closed
             (end (progn (search-forward-regexp
                          "\\\\end{notate}")
                         (match-beginning 0))))
        (when (not (equal tag ""))
          (scholium 'Tag-to-Name-converter
                    (add-to-or-start-list
                     (scholium-text (get-article
                                     'Tag-to-Name-converter))
                     `(,tag . ,name))))
        (scholium name
                  (buffer-substring-no-properties beg end)
                  `((,current-parent parent))
                  '(note))
        (setq notes
              (append notes (list name)))
        (import-code-continuations)))
    notes))
\end{lisp}

\begin{notate}{On `import-code-continuations'}
This runs within the scope of `import-notes', to turn any Lisp chunks
that follow a given Note into scholia attached to that note.  (This
won't import the one stand-alone ``verbatim'' environment used in the
code; we could change this function, or include the environment inside
of a Note.  Or we could just put the literate programming code into
another file.  Minor issue.  Also, figure environments would be left
out, except for a small trick, namely including them within
surrounding Notes.)  Notice that previous versions of these articles
shouldn't exist at import time, or Lisp sections will be doubled.

The code continuations and the code descriptions wind up being
mutually about one another, which is kind of cute.
\end{notate}

\begin{lisp}
(defun import-code-continuations ()
  ;; ugly formatting = latex overrides!
  (while (looking-at (concat "\n\n\\\\b" "egin{lisp}"))
    (scholium `(code-continuation ,name)
              (let ((old-text (scholium-text
                               (get-article
                                `(code-continuation ,name))))
                    (new-text (buffer-substring-no-properties
                               (progn (next-line 3)
                                      (line-beginning-position))
                               (progn (search-forward-regexp
                                       (concat "\\\\e" "nd{lisp}"))
                                      (match-beginning 0)))))
                (if old-text 
                    (concat old-text "\n" new-text)
                  new-text))
              `(((article ,name)))
              'code)
    ;; this should add an appropriate link to the article
    ;; that this is a code-continuation of.
    (let ((article-to-edit (get-article name)))
      (scholium (scholium-name article-to-edit)
                (scholium-text article-to-edit)
                (add-to-or-start-list 
                 (scholium-about article-to-edit)
                 `(code-continuation (code-continuation ,name)))
                (scholium-bookkeeping article-to-edit)))))
\end{lisp}

\begin{notate}{A theoretical issue with code continuations}
I like to think of the notes as being scholia
attached to the code, rather than the code as scholia attached
to the notes.  But the way the importing routine is set up,
this relationship is reversed.
\end{notate}

\begin{notate}{Rendering code continuations}
If we know what kind of code is stored in the code continuation
articles, we may want to render it in the appropriate mode.  (The use
of connection coloration is disruptive to this kind of display, so
we'd want to use some other mechanism for showing associations, if
these need to be shown.  See Note \ref{color-and-style}.)
\end{notate}

\begin{notate}{Identification of notes is slow}
I tried identifying each note with its source region by running
\begin{idea}
(add-to-scholia-property-within-region
 beg
 end
 `(identifies-with ,name))
\end{idea}
towards the end of `import-notes', but it slowed everything down.

We may be able to get something similar by other means
anyway (Note \ref{alternative-to-identification}).
\end{notate}

\begin{lisp}
(defun browse-scholium-system ()
  (interactive)
  (import-scholium-system)
  (display-article 'section)
  (message "You've successfully imported the system!"))
\end{lisp}

\begin{notate}{Finding all articles with code continuations} \label{with-code}
Cf. Note \ref{article-menu-list-articles-matching-predicate}.
\begin{idea}
(article-menu-list-articles-matching-predicate
 (lambda (name)
   (let ((article (get-article name)))
     (and (member-if (lambda (link)
                       (typedata-includes (cdr link) 
                                          'code-continuation))
                     (scholium-about article))
          (typedata-includes (scholium-type article)
                             'note)))))
\end{idea}
You can then run `listing-to-compilation' to get a new document that
drops all the interstitial chit chat found in this one.
\end{notate}

\begin{notate}{Finding all articles without code continuations} \label{without-code}
For our friends from the liberal arts,
\begin{idea}
(article-menu-list-articles-matching-predicate
 (lambda (name)
   (let ((article (get-article name)))
     (and (not (member-if (lambda (link)
                            (typedata-includes (cdr link) 
                                               'code-continuation))
                          (scholium-about article)))
          (typedata-includes (scholium-type article)
                             'note)))))
\end{idea}

(Curiously enough, this document currently seems pretty balanced in
that the code plus immediately important documentation takes up just
about the same amount of space as the philosophical musings.  Well,
almost as much.  And of course those verbatim environments are long.)
\end{notate}

\begin{notate}{Finding all articles with emphasized parenthetical remarks}
Cf. Note \ref{article-menu-list-articles-matching-regexp}.  These are
the articles that contain side remarks that should eventually be turned
into scholia.
\begin{idea}
(progn (article-menu-list-articles-matching-regexp "\\\\emph{(")
       (listing-to-label "Comments"))
\end{idea}
\end{notate}

\begin{notate}{Exporting LaTeX docs} \label{exporting-latex-docs}
This should give an inverse to importing
\LaTeX\ documents (Note \ref{importing-latex-docs}).
Exporting in one go is a cheap alternative to live updating via
identifications (Section \ref{identification}).  Either would
work, but exporting seems a little easier to get going quickly.
(Incidentally, exporting is similar to producing a compilation;
see Section \ref{quick-compilations}.)
\end{notate}

\begin{notate}{On `export-scholium-system'} \label{export-scholium-system}
We use a similar sort of "recursive" style to that used in the import.
First, loop through the sections.  These must of course be exported in
their own limited fashion.  Then, for each, examine the contents. The
items contained in each section are either subsections or notes.  If
the item really is a subsection, then do something similar to what we
did with sections; otherwise, it is a note and we simply export it.
The items at the penultimate level are either subsubsections or notes;
and finally, in the end, the items are all notes.

Note that getting the ``Tag-to-Name-converter'' each time it
is used in the child functions is really sort of excessive,
since it could be found once in the parent.  However, things seem
to run fast enough. 
\end{notate}

\begin{lisp}
(defun export-scholium-system ()
  (set-buffer (get-buffer-create "*Export*"))
  (erase-buffer)
  (insert (scholium-text (get-article "Scholium system frontmatter")))
  (dolist (sect (scholium-text (get-article 'section)))
    (export-section sect)
    (let ((contents (scholium-text (get-article sect))))
      (dolist (item contents)
        (let ((current-item (get-article item)))
          (if (typedata-includes (scholium-type current-item)
                                 'subsection)
              (export-subsection (scholium-name current-item))
            (export-note current-item))))))
  (insert (scholium-text (get-article "Scholium system backmatter"))))
\end{lisp}

\begin{notate}{Child functions} \label{child-functions}
Notice the sort of interesting use of the words ``child'' and
``parent'' in Note \ref{export-scholium-system}; this is what comes
of working on section listings most of the day.  No reason that I can
think of not to view code this way.  Eventually we may want to parse
Lisp functions and annotate them with hierarchies representing this
sort of information.
\end{notate}

\begin{notate}{Stable import/export}
The text, import, and export routines are all adapted to each other
well enough to make it so that the text is stable under importing and
subsequent exporting.  We'll have to do more extensive testing to be
sure that it is also behaving properly with regard to intervening
editing operations, but things are currently looking pretty good in
this regard.  (But be careful, and back up your work!)
\end{notate}

\begin{notate}{On `export-note'} \label{export-note}
This is for exporting notes.  When the note has a code
continuation it also exports that code.

To really work, this is going to have to replace references with
\verb|\refs| and \verb|\refs|.  Hopefully we're storing the relevant
information somewhere easily accessible at import time?  We should at
very least be able to do the opposite of what is done with
`swap-out-latex-references'.

We essentially need to render the article to know
what the relevant references are (and where they are).
This suggests to me that if we don't want to disrupt the
user's experience of the rendered articles appearing in
the standard display, we should be able to ``beam'' a
rendering to some other buffer and then pick it up from
there.  Also, if some references have been generated
that don't correspond to items with tags, we'll have to
accomodate that.
\end{notate}

\begin{lisp}
(defun export-note (note)
  (set-buffer (get-buffer "*Export*"))
  (let* ((name (scholium-name note))
         (tag (car (rassoc name (scholium-text
                                 (get-article
                                  'Tag-to-Name-converter))))))
    (insert "\\begin{notate}{" name "}"
            (if tag 
                (concat " \\label{" tag "}\n")
              "\n")
            (swap-in-latex-references note)
            "\\end{notate}\n\n")
    (dolist (scholium (mapcar (lambda (backlink)
                                    (get-article (car backlink)))
                              (get-backlinks name)))
      ;; we were missing one function from a list of two in my last
      ;; test of exporting, should look into this.
      (when (typedata-includes (scholium-type scholium) 'code)
        (insert "\\b" "egin{lisp}\n"
                (scholium-text scholium)
                "\\e" "nd{lisp}\n\n")))))
\end{lisp}

\begin{notate}{On `swap-in-latex-references'} \label{swap-in-latex-references}
Maybe it would be easier if each of the references had
the name of the tag stored locally?  This doesn't
seem like it would actually help.  The point I guess
is that we don't want to detect the scholia structure
all at once, but rather, zip through and make changes
to each item we encounter, in order.  I.e., the endpoints
of the regions that we'll be modifying are always in
transition.

`move-to-next-region-with-scholium' will have to be set up to
return `nil' if there is no next scholium.  It would also be
nice if this returned the boundaries of the region to which the
scholium is attached, when there is a scholium, since we're
going to want to do a replacement of the text in that region.
(If it proves more reasonable to use a different function for
this purpose, that would be fine, but this is roughly what it
should do.)  Note that `move-to-next-region-with-scholium'
currently works in terms of overlays, which isn't really
right.

After we find the scholia at the start and end of the marked region,
we could check a complicated condition to see if there is a reference
at the beginning and another copy of the same reference at the end.
But for the time being we aren't going to work with this general case
of overlapping references, and we'll just assume one reference per
marked region and get on with things.
\end{notate}

\begin{lisp}
(defun swap-in-latex-references (note)
  (save-excursion
    (let ((tags-alist (scholium-text (get-article
                                      'Tag-to-Name-converter))))
      (display-article (scholium-name note))
      (set-buffer (get-buffer "Main Article Display"))
      (let ((next-region (move-to-next-region-with-scholium)))
        (while next-region
          (let ((scholium-property-at-start 
                 (get-text-property (first next-region) 'scholia))
                (scholium-property-at-end 
                 (get-text-property (second next-region) 'scholia)))
            (let* ((scholium-id (first scholium-property-at-start))
                   (scholium-name (first scholium-id))
                   (possible-reference (get-article scholium-name)))
              (when (typedata-includes 
                     (scholium-type possible-reference)
                     'reference)
                (let ((scholium-tag 
                       (car (rassoc (reference-to-article 
                                     possible-reference)
                                    tags-alist))))
                  (if scholium-tag
                      (progn (delete-region (first next-region)
                                            (second next-region))
                             (goto-char (first next-region))
                             (insert "\\ref{" scholium-tag "}"))
                    (let ((new-tag (replace-regexp-in-string
                                    " " "-"
                                    (buffer-substring-no-properties
                                     (first next-region)
                                     (second next-region)))))
                      (delete-region (first next-region)
                                     (second next-region))
                      (goto-char (first next-region))
                      (insert "\\ref{" new-tag "}"))))))
            (setq next-region (move-to-next-region-with-scholium)))))
      (buffer-substring-no-properties (point-min)
                                      (point-max)))))

(defun export-section (section-name)
  (set-buffer (get-buffer "*Export*"))
  (let* ((tag (car (rassoc section-name (scholium-text
                                         (get-article
                                          'Tag-to-Name-converter))))))
    (insert "\\section{" section-name "}"
            (if tag 
                (concat " \\label{" tag "}\n\n")
              "\n\n"))))

(defun export-subsection (subsection-name)
  (set-buffer (get-buffer "*Export*"))
  (let* ((tag (car (rassoc subsection-name (scholium-text
                                            (get-article
                                             'Tag-to-Name-converter))))))
    (insert "\\subsection{" subsection-name "}"
            (if tag 
                (concat " \\label{" tag "}\n\n")
              "\n\n")))
  (let ((contents (scholium-text current-item)))
    (dolist (item contents)
      (let ((current-item (get-article item)))
        (if (typedata-includes (scholium-type current-item)
                               'subsubsection)
            (export-subsubsection (scholium-name current-item))
          (export-note current-item))))))

(defun export-subsubsection (subsubsection-name)
  (set-buffer (get-buffer "*Export*"))
  (let* ((tag (car (rassoc subsubsection-name (scholium-text
                                               (get-article
                                                'Tag-to-Name-converter))))))
    (insert "\\subsubsection{" subsubsection-name "}"
            (if tag 
                (concat " \\label{" tag "}\n\n")
              "\n\n")))
  (let ((contents (scholium-text current-item)))
                              
    (dolist (item contents)
      (let ((current-item (get-article item)))
        (export-note current-item)))))
\end{lisp}

\begin{notate}{Name conflicts}
As we talked about vis a vis namespaces (Note
\ref{labels-and-namespaces}), it would be nice if
we could deal with multiple different things having
the same name, say a Section and a Note, or two
Notes in different sections.  A disambiguation page
could be created whenever the ambiguity needed to be
paid attention to.
\end{notate}

\begin{notate}{Revise to support ``function'' type articles} \label{elisp-function-type}
Instead of referring to Notes that describe functions, sometimes
we may prefer to refer to ``function'' articles, which will
include the function and the note that describes it.
\end{notate}

\begin{notate}{Group functions sharing scope}
Objects with type functions could have ``runs
within the scope of this function'' slots.  This would facilitate
propagating changes in notation.
\end{notate}

\begin{notate}{Other features for function articles}
As top-level objects, functions could be fitted
out with their own fora and FAQ's.  They could be
displayed with the appropriate mode turned on
by default.

A short document containing only the functions that
one had been working on recently could be created easily;
see Note \ref{making-a-compilation} and Note \ref{recent-referents}.
\end{notate}

\subsection{On-the-fly scholium creation} \label{otf}

\begin{notate}{Why on-the-fly?}
This section is about creating scholia automatically.  For example,
every time I type ``scholium system'', maybe something happens -- for
instance, maybe the text gets marked up with a reference to some other
object describing the scholium system.  This is sort of like what
happens in No\"osphere, though there, auto-linking happens when the
document is rendered.

Another similar thing is the auto-indexing feature present in Apple
Computer, Inc.'s latest OS, wherein content is automatically indexed
as it is being created.  We could do something fairly similar with the
scholium system (and some of this indexed information could be shared
with other collaborators whenever there are multiple people using the
scholium system together).

Another example use would be to display interesting feedback about
things you had just typed -- maybe related definitions or phrases,
or some other cool stuff.  Doing this on an automatic on the fly
basis is a minor simplification (from the user's point of view)
over doing it only when the user issues an explicit command.

Obviously you don't want to search every editspace.  For example,
displaying a \emph{definition} as a scholium means that you would
probably just want to search the dictionary namespace.  This is
\emph{not} the same as creating a scholium, indeed, the dictionary may
not ``really'' contain scholia at all; you can use the structure of
the dictionary to easily retrieve information that you would like to
superimpose on the current document.  (This is a somewhat traditional
sounding digital libraries sort of thing.)
\end{notate}

\begin{notate}{Compare flyspell}
Some good ideas might be available in the flyspell package for Emacs.
\end{notate}

\begin{notate}{Automatically adding files to the article list}
We can also set things up so that visited files are added to the
article list automatically, etc.
\end{notate}

\begin{notate}{Lexical autolinking}
One major application of this system would be to create
No\"osphere-like autolinking.  Thus, part of this ``experiment'' is to
reimplement autolinking in this model.  Note, that once you have
implemented linking within the model, then autolinking should probably
be ``easy'' to add by reimplementing the No\"osphere algorithm.  One
advantage of doing things this way is that links appear at edit time
rather than render time (presumably anyway), which means that they can
be tinkered with right away as needed.

I wonder if the autolinker could be used to make an index of terms
automatically, too?  Seems likely.
\end{notate}

\subsection{Simulating a wiki with the scholium system} \label{wiki}

\begin{notate}{Build scholia by first adding markup and then parsing}
The idea in this section is to assemble a collection of scholia that
simulate the page-reference markup found in a wiki.  The strategy we
use is to read the wiki page and add \emph{text properties} to the
``alt text'' indicating the link targets.  We then parse the buffer
and create scholia about the regions that bear text properties.  The
reason for going through this two-step process is that the position of
text in the buffer will be shifting around as we get rid of wiki
markup, so it is convenient to add scholium system markup in a format
that in invariant under such shifting.  To the extent that we need to
specify \inp{type} data, we overload the \inp{name} data that we use
for initial markup.  For example, ``(external-page <page-name>
<URL>)'' is the format we'd choose as the \inp{name} of scholia
representing references to external pages.
\end{notate}

\begin{notate}{On `mark-up-wiki-with-reference-scholia'}
Find things in double brackets, possibly with an intervening ``|'',
making sure to find the minimal double bracket pair.  What
\emph{should} happen (though the current draft is a bit different) is:
a reference scholium should be created for each reference, and stuck into the
article list in some appropriate place (presumably the namespace
associated with the article in question, more specifically, on some
specific reference subspace of that space).  \emph{Then} when someone goes
to render the document in question, we grab the references out of their
storage facility and mark up the document with text properties.
(The current implementation is really just a regexp check...)
\end{notate}

\begin{lisp}
(defun mark-up-wiki-with-reference-scholia ()
  (save-excursion
    (goto-char (point-min))
    (while (re-search-forward
            ;; we could probably get all types of reference in
            ;; one regexp.  I'm not sure why the old
            ;; version for grabbing external references was deleted,
            ;; but it is featured in the Sept 4 printout.
            "\\[\\[\\([^]|\n]+\\)\\(|\\)?\\([^]]+?\\)?]]" nil t)
      (replace-match (propertize (or (match-string 3)
                                     (match-string 1))
                                 'scholia
                                 "Reference")))))
\end{lisp}

\begin{notate}{Recent changes and recent referents} \label{recent-referents}
It would be handy to have some code that would list the articles that
have had new scholia written about them.  This would help a user (e.g.
the author of the articles under discussion) see what people are
talking about.  Some additional semantics could be provided, to list
recent modified versions, which are similar to document changes, but
arguably more relevant in a system with object ownership.  Actual
changes should be listed too.
\end{notate}

\begin{notate}{Handling references}
(1) Do special markup;
(2) What is the format of a reference precisely?;
(3) Do they have backlinks?;
(4) Notice that they will probably need to have a special markup
property associated with them (instead of the typical `scholia'
property);
(5) Note that we might want to record all the references associated with
a given buffer on its metadata article, as well as on the full list
of articles and a `reference' label (unless we were to just make a reference
namespace).
\end{notate}

\begin{notate}{Strikethrough}
One of the most important things you can do with a wiki
is to delete things that other people have written.
Showing ``deletion'' as a sort of scholium would be
a handy feature for a wiki modeling experiment.
\end{notate}

\begin{notate}{Delete/insert patches}
Note that patches that delete or insert changes are something like
``quoted'' changes to files; i.e., as opposed to the ``direct''
changes that occur when editing.
\end{notate}

\subsection{Using the scholium system to do AI stuff} \label{AI-stuff}

\begin{notate}{Take an example from a known system}
It would be a good idea to find at least one example from a ``standard
AI'' system (KM's chemistry-problem solving, for example) and import
and otherwise emulate it in the system here.  We'll have to look
around for a good example.
\end{notate}

\subsection{Using the scholium system for HDM things} \label{HDM-things}

\begin{notate}{What is HDM and how does it relate to the scholium system?}
The HDM (expanded: Hyperreal Dictionary of Mathematics) project is
about \emph{taking the world's mathematical knowledge, and getting it
  into a format that is as useful as possible.}

The key point to make here is that HDM will be a free culture artifact
containing lots and lots of mathematical information (definitions,
theorems, proofs, expositions, examples, and excursions, etc.) that
will need to be organized for use by both individual humans and
networks of machines.  The scholium system is part of the HDM project
because of these organizational needs -- \emph{and} (in particular) it
is being designed in such a way that it should be able to satisfy the
``AI needs'' of the project.

The revolution will be computerized -- and it will be total.
\end{notate}

\begin{notate}{Application of the scholium principle for HDM}
We should talk about encoding different types of objects in the system
and showing their relationships using typed links; also about
assembling reports via transclusion, and so on and so forth.  We
should also show how this system can be used as a stand-in for more
``traditional'' KRR systems (SNePs, KM), and how reasoning modules can
be implemented for the system.  But this is pretty advanced stuff, and
it is going to be a little while until we get to it.

In the mean time, we have various fun things (a little music and even
less Go), and we can think about the relationships to Dirk Gently and
so on and so forth.
\end{notate}

\begin{notate}{Template utilities as an example}
Ray was working on a set of template utilities that could be encoded
as a scholium-based document.
\end{notate}

\begin{notate}{Overview of Arxana as an example}
It might be nice to create a simplified version of the current
document that would deal with the core functions and ideas only.  This
would presumably help people learn how to work with the system.
\end{notate}

\subsubsection{Managing the contents of a mathematical dictionary}

\begin{notate}{Importing the APM-$Xi$}
See \ref{import-apmxi} (currently the main function in this section,
although we may come up with some additional functions later on).
Actually, we should probably also import the APM, however, I'm not
completely sure I can get copyright waivers for the APM's content.  I
hope so; I should investigate that.  If I can get written permission,
then I suppose we should put APM into the HDM's archive along with
APM-$\Xi$.
\end{notate}

\begin{notate}{Large-scale structure of APM and APM-$\Xi$}
We have some words and phrases.  These phrases come up in the APM in
various places.  Each phrase has a list of problems that it is
associated with.  It also has a list of definitions that it is
associated with.  For APM, we would want the relevant definitions to
appear as attached scholia.  For APM-$\Xi$, we presumably want a
namespace for each phrase, into which we put each of the definitions
as its own article, together with the specially-designated list of
associated problems.  Here it seems like we have the opportunity to
use links with different types.  (Maybe we would just attach the list
of associated problems to the namespace as a whole.)  To parse the
document this far, the first thing to do is to break it into sections,
then parse each of these sections into lists of problems.
\end{notate}

\begin{notate}{Types for the APM-$\Xi$}
We keep track of the sections, entries, and stubs that belong to
APM-$\Xi$ as special types.
\end{notate}

\begin{lisp}
(add-to-list 'modified-type-labels '(apmxi-section . apmxi-section) t)
(add-to-list 'modified-type-labels '(apmxi-entry . apmxi-entry) t)
(add-to-list 'modified-type-labels '(apmxi-stub . apmxi-stub) t)
\end{lisp}

\begin{notate}{On `make-apmxi-sections-into-articles'}
This function makes every section in the APM-$\Xi$ into its own
article.  The list of these articles is recorded on the article
`apmxi-section'.
\end{notate}

\begin{lisp}
(defun make-apmxi-sections-into-articles ()
  (save-window-excursion
    (find-file "../p2/Xi.tex")
    (goto-char (point-min))
    (while (re-search-forward "^\\\\section{{ \\(.+\\)}} " nil t)
      (let ((article-name (match-string-no-properties 1))
            (beg (point))
            (end (save-excursion (search-forward "\\section" nil t)
                                 (match-beginning 0))))
        (when end
          (scholium article-name
                    (buffer-substring-no-properties beg end)
                    nil
                    'apmxi-section))))))
\end{lisp}

\begin{notate}{On `chunk-out-apmxi-definitions'} \label{chunk-out-apmxi-definitions}
Apparently some of the terms in APM-$\Xi$ were defined in multiple
sections.  At least, \emph{stubs} appear in multiple places.  This
should presumably create multiple definitions when multiple
definitions are given, or get rid of stub-like definitions, or
something.

Note that the way this is set up, even sections that contain only
stubs can also contain references to the real articles, when they
exist -- this means that the real articles may appear in several
listings, which I think is as it should be.
\end{notate}

\begin{lisp}
(defun chunk-out-apmxi-definitions ()
  (map-label (lambda (name)
               (with-temp-buffer
                 ;; this ensures that sexps are defined properly
                 (latex-mode)
                 (let ((topic (get-article name))
                       contents)
                   (insert (scholium-text topic))
                   (goto-char (point-min))
                   (while (re-search-forward "^(" nil t)
                     (let* ((beg (match-beginning 0))
                            (title-beg (match-end 0))
                            (title-end (progn (goto-char beg)
                                              (forward-sexp)
                                              (1- (point))))
                            (stub-p (looking-at " *$"))
                            (title (buffer-substring-no-properties
                                    title-beg
                                    title-end))
                            (end (save-excursion
                                   (search-forward-regexp "^$")))
                            (possible-previous (get-article title)))
                       (when (or (not possible-previous)
                                 (typedata-includes (scholium-type
                                                     possible-previous)
                                                    'apmxi-stub))
                         (scholium title
                                   (buffer-substring-no-properties
                                    beg
                                    end)
                                   `((,name parent))
                                   (if stub-p
                                       '(apmxi-stub apmxi-entry)
                                     'apmxi-entry)))
                       (setq contents (cons title contents))))
                   (scholium name
                             contents
                             nil
                             '(apmxi-section label)))))
             'apmxi-section))
\end{lisp}

\begin{notate}{On `import-apmxi'} \label{import-apmxi}
This function puts into play all of the other functions developed
above, to fully process the APM-$\Xi$ and get it nicely situated in
the scholium system.
\end{notate}

\begin{lisp}
(defun import-apmxi ()
  (make-apmxi-sections-into-articles)
  (chunk-out-apmxi-definitions))
\end{lisp}

\begin{notate}{Special reference types} \label{special-reference-types}
In the mathematics domain, would be handy to have ``related to'',
``depends on'', ``proof of'' and ``prerequisite for understanding''
reference sub-types.  It is easy enough to create a subtype -- just
add the secondary category to the \inp{type} variable along with the
``reference'' datum.  Special types of references are one of the key
semantic network-like features of the scholium system (see also Note
\ref{type-creation}).  Maybe the reference-creating facility should
provide an optional second argument for specifying reference sub-type.
\end{notate}

\begin{notate}{Selecting all APM-$\Xi$ stubs}
If you want to look at a list of all stubs in the APM-$\Xi$, run this code:
\begin{idea}
(article-menu-list-articles-matching-predicate 
 (lambda (name)
   (when (typedata-includes (scholium-type (get-article name))
                            'apmxi-stub)
     t)))
\end{idea}
(See  Note \ref{article-menu-list-articles-matching-predicate}.)
\end{notate}

\begin{notate}{Screening out APM-$\Xi$ stubs} \label{apmxi-stub-processing}
If you want to screen \emph{out} the stubs in the APM-$\Xi$, run
\begin{idea}
(display-difference-of-labels 'apmxi-entry 'apmxi-stub)
\end{idea}
(see Note \ref{display-difference-of-labels}).  Of course,
eventually we're going to want to do some more processing
to make the stubs into useful things.
\end{notate}

\subsubsection{Facilitate collaboration between HDM authors}

\begin{notate}{Use a simple scheme}
The collaboration scheme is described in Section
\ref{multiuser-cooperation}.  I might try to make that version
``general'' and in this section implement some ``specifics''.
\end{notate}

\subsection{Model a physical space and its contents} \label{space-model}
\begin{notate}{Holtzermann Kitchen Companion}
I've been thinking that it would be handy to have a full inventory
of the kitchen I share.  The inventory, or model, really, could be
shared with the kitchen-mates and it could help us remember where
things go and what needs to happen around the place.  More generally,
the idea would be: to help improve communication about goings on in
this shared space.
\end{notate}

\begin{notate}{Typelock} \label{typelock}
In order to talk about the contents of cupboards versus the
refrigerator, for example, the user should be able to specify
a bunch of `(contains A B)' relationships.  Both `contains'
and `A' will stay fixed for reasonably long periods of time.

In addition, probably everything that is created in this experiment
should have a {\tt kitchen-element} type (and label).  Thus,
we want some sort of typelock at the level of object creation
and link creation.  (Indeed, we could probably set things
up so that the links didn't need to be specially-typed, but
rather were just part of some specially-typed scholia.)

A sort of segue away from the main topic: it could be handy
to optionally display all of the `contains' items using the listing
mechanism instead of the standard scholia display.
\end{notate}

\begin{notate}{Case for namespaces}
% Let it not be said that this document contains everything but the
% kitchen sink.  Here is the sink.
Both the sink cabinet and the doorless cupboard contain a ``drawerful
of assorted utensils''.  Of course, these aren't \emph{the same}
drawers (nor are they the same utensils).  We find the same sort of
thing in earlier imports of the scholium system, in which several
sections had ``Further notes'' subsections.  As a work-around, all of
these were given distinct names, but subsequently, especially when
working with complexes of formulaic scholia (Note \ref{formulaic}),
that behavior wouldn't be desirable.
\end{notate}

\subsection{Make more physical scholia} \label{more-physical-scholia}

\begin{notate}{Extended rendering}
The scholium-based document idea can be extended outside of the
computer.  The examples that are produced this way can help illuminate
and inspire out work.  See Note \ref{physical-scholium-system}.
\end{notate}

\begin{notate}{Math paper as gallery tour} \label{math-paper-as-gallery-tour}
We could put pictures from a math paper on the walls of a gallery,
with the space divided up in such a way as to have each wing
correspond to a section of the paper.
\end{notate}

\begin{notate}{Scholia on architectural space}
A generalization of Note \ref{math-paper-as-gallery-tour} is to put
together spaces that incorporate scholia into their surfaces
(graffiti, chalkboards), or into their basic design.  Memory spaces
(mnemonic architectures) are spaces with double (or more) meanings,
i.e., multiple sets of scholia on the same space.
\end{notate}

\begin{notate}{The HPDA}
This is a somewhat famous example (among a certain geek crowd) example
of a scholium-based document system.  I invented a game that goes
along with it.
\end{notate}

\begin{notate}{Printing things out}
Questions as to the impact and relevance of putting things on paper
continue.
\end{notate}

\begin{notate}{Node-based systems}
Systems like freemind\footnote{{\tt
    https://sourceforge.net/projects/freemind }} aren't exactly
``physical'' in the sense of something you can handle, but they
provide a different sort of view into a document, and views are
physical things.  (It seems to that freemind itself only provides
hierarchical views, which isn't particularly pleasant for us.  I know
that the KM people had a tool that looked good for more complicated
views, but I'm also pretty sure they didn't want to release this tool
to the public.)  Some sort of nice graphical tool could be used to
make a schematic of the scholium system that people might find useful.
\end{notate}

\begin{notate}{Outline management}
In Table \ref{scheme-for-outline-managemnent}, we illustrate
line-numbers 1--5 at left, items a--e at right, together with labels
$\alpha$--$\epsilon$.  We envision this outline changing over time,
and references such as \verb|\itemref{|$\alpha$\verb|}| consistently
producing the correct line number.  I think \LaTeX\ can already handle
this sort of thing, I just forget how you do it.
\begin{table}
\begin{center}
\begin{tabular}{ll}
1 & a ($\alpha$) \\
2 & \quad b ($\beta$) \\
3 & c ($\gamma$) \\
4 & \quad d ($\delta$) \\
5 & \quad \quad e ($\epsilon$) 
\end{tabular}
\end{center}
\caption{A scheme for outline management \label{scheme-for-outline-managemnent}}
\end{table}
\end{notate}

\begin{notate}{Offline usefulness} \label{online-usefulness}
Printed documents are only useful offline; the question as to whether
they are more useful offline than a non-networked computer remains.
Sometimes they will have a certain pragmatic edge, e.g., they allow
one to avoid lugging the computer from place to place.  On the other
hand, one typically then must type up one's handwritten comments.

(The problem of integrating interesting documents into Arxana is
present whether or not these documents are physical.)
\end{notate}

\begin{notate}{Getting jobs done in different ways}
Many of the things we talk about in Arxana (and in the HDM project in
general) have to do with finding new ways to get jobs done which we
already can get done in some other way.  The hope is that the new ways
of working will allow us to be more efficient or have more fun or that
they will otherwise be beneficial.  (It isn't completely clear that
you really do get the same job done if you do it in a different way.)
\end{notate}

\section{Philosophy of the scholium system} \label{philosophy}

\begin{notate}{Code} \label{code}
The concerns mentioned in Note \ref{the-philosophy} are closely
related to the questions and ideas of ``Code and Other Laws of
Cyberspace''\footnote{{\tt http://codebook.jot.com/WikiHome}}.
Cf. Note \ref{consistency-why}.

I'd be interested to know whether pass-by-reference as a basis for
document exchange doesn't provide a final solution to the copyright
problem.  OK, almost final; you still need some high-tech system for
managing the references and finally retrieving and operating on
content.
\end{notate}

\begin{notate}{An observation} \label{observation-on-cyberspace}
There is a lot of room in cyberspace.
\end{notate}

\begin{notate}{Definitions of freedom}
In Note \ref{inspirations}, I talk about the importance of user
freedom.  I should go into this more.  For now, I'll just say that for
me ``user freedom'' in some sense includes the notion of ``user
convenience.''  Why?  Because what is \emph{freedom to do} if it
doesn't go along with actual \emph{ability to do}?  \emph{Empty!}  I
call the basic notion at work here the ``liberties model of freedom.''
(With all due apologies to The Dead Kennedys.)

(Note that the guaranteed freedom to do something that one might
potentially be able to do at some point but can't do just now isn't
empty, it's more like ``half full''.)
\end{notate}

\begin{notate}{``Free culture'' is ambiguous} \label{free-culture}
We should be careful when using this term to make sure it is really
defined properly, something I don't think anyone really has done yet.
I'm not really cool with using terms that aren't defined.  But then I
get caught up using the term.  So, I'm not sure.  I'm guessing that
there is a false dichotomy here similar to the dichotomy between
``text'' and ``hypertext'' -- in other words, something that could
perhaps be formalized, but which as yet hasn't been (see Note
\ref{hyperstuff}).  My sense is that ``free culture'' may be as
ambiguously overloaded a term as ``intellectual property'' (just for
example).  And yet, freedom seems to be an important part of a useful
system.  We proceed with caution.
\end{notate}

\begin{notate}{``All information is superimposed.''}
Statements are superimposed on an author's perceptions and inner life.
Perceptions are superimposed on the base data of experience (and are a
part of this data).  Everything we identify as ``information'' is
superimposed.
\end{notate}

\begin{notate}{What, then is at the bottom?}
Is there a bottom, a ``real'' reality, even if it is an
extra-informational one?  Or is reality as we know it a mutually
agreed-upon set of mutually-referential superimposed bits?  (And for
that matter, is there a top, and what would \emph{that} look like?)
\end{notate}

\begin{notate}{A possible interpretation of physical scholium systems}
The physical systems of Section \ref{more-physical-scholia} and
elsewhere show that physical systems can ``do'' simulation.  Compare
also Alan Moore's ``Supreme'', a fictional character who is somewhat
aware of the effects of being as such, although not \emph{quite} to
the point of being fully cognizant of the fact that he is fictional.
Physical scholium systems are only partially aware of their
computational possibilities (and, mutatis mutandi, computational
systems don't always fully take into account their physical
manifestations, whether concrete or only barely imaginable).
\end{notate}

\begin{notate}{Making things easier to learn}
If subject matter to be learned is embedded in a scholium system,
questions and answers that people have asked already would tend to be
easily accessible and clearly organized.  It would also be easy to ask
new questions in the context in which the right people will take
notice.
\end{notate}

\begin{notate}{The extensions of man}
What are the things you can do in a given medium?  What can you point
at (indicate)?  What sorts of gestures can you make?  See Note
\ref{human-and-artificial-intelligence}.
\end{notate}

\begin{notate}{Designing a free code market}
A ``consortium'' of free software supporting companies would be
similar to the scholium model of business and relates to FSB.  It is
important to know what is in demand and what the supply and costs are.
\end{notate}

\begin{notate}{Scholia in ``The star's my destination''}
The modifications to the main character's teeth are an example of
actionable scholia.  The tattoos are another obvious example, adding
art on top of skin.  This makes me think that perhaps the issue of
scholia also comes up in the book by Dumas that this one draws so
heavily upon.  The SF book might also be considered to be a sort of
scholium attached to Dumas's ``The Count of Monty Cristo'' (cf. Note
\ref{barkovs-hamlet}).
\end{notate}

\begin{notate}{Anthropogenic change}
Changes effected by humans upon the world might be considered to be
scholia.  Changes effected upon ourselves (whether skin-deep or
destiny-changing) could be as well.  But this vision of anthropogenic
change is also somewhat \emph{anthrocentric}.  Humans are just part of
a much broader world.  Our ``will'' and ``agency'' are brought into
question in the context in which ``everything is connected.''
(Compare ``Les Enfants Terrible'' on that one.)
\end{notate}

\begin{notate}{Document closure} \label{closure}
Wittgenstein said that the world is all that is the case.  Baudrillard
said that the world is that which can't be exchanged for anything
else.  A document that contains references to things outside of itself
is really an article in a larger document.
\end{notate}

\begin{notate}{When is a document finished?}
The question as to what needs to go into a document before the
document is finished has probably been thought about by many authors
and editors in the past.  Some things are bound to be off topic,
whereas other things are bound to be completely critical.  Some things
are of only peripheral importance.
\end{notate}

\begin{notate}{Different kinds of documents}
Different kinds of documents correspond to different sorts of context
-- spatial, temporal, ideational, etc. (see Note \ref{context}).
Note \ref{container}, Note \ref{thread}, Note \ref{process}, and
Note \ref{collection} talk about several different kinds of
documents.  See also Note \ref{displaying-nearby-scholia} for some
further words about different kinds of context.
\end{notate}

\begin{notate}{Document as container} \label{container}
Each document or article contains statements.  Sometimes these
statements can be bonafide objects, for example, each section in this
document contains Notes and certain other objects.  ``Frames'' and
``ontologies'' fall into this category (and presumably so does the
notion of ``category'').  This model seems particularly well-adapted
to the notion of transclusion (and vice versa), and to the idea of a
document as the expansion of an outline.  Comments about the document
as a whole should apply to the containing object (so that in some
cases, comments about the document as a whole come from within, i.e.,
comments need to be contained somewhere; see Note \ref{closure}).  If
we set the current document up in the scholium system on public
display (and for public comment), we would probably want to have a
special place for general comments about the article.  This relates to
the issue of namespaces and labels (Note
\ref{labels-and-namespaces}).
\end{notate}

\begin{notate}{Document as thread} \label{thread}
Sometimes a document takes you along a linear path through a more
complicated space (see Note
\ref{transclusion-induces-linear-structure} for a description of one
way this can work).  Frequently authors don't know where their writing
is going when they start.  Franz Kafka and Stephen King are notable
proponents of this style of writing, which is almost the antithesis of
writing from an outline.  Whether stitching a specific pattern or
exploring a strange labyrinth, the writer's experience has several
threadlike qualities.  When there are many authors, then the metaphor
is more obviously one of weaving many threads.  Of course, individual
threads and an overall threadlike quality writing only appear under
certain circumstances.  (I wonder what Dijkstra said about writing.
One thing I know is that his thought about writing proofs was that
each step should be made clear; the reader shouldn't be left
guessing.)
\end{notate}

\begin{notate}{Document as process} \label{process}
The view advanced in Note \ref{thread} admits, but does not
necessarily imply, the view of the document as a \emph{process}.  A
thread or body of woven threads could exist outside of the process
proper; in this model, the author might follow the thread along to its
conclusion (like a detective ``unraveling'' a mystery).  It seems
apparent that a document, whether it is written by one or many authors
is a process, at least until it is ``finished''.  Even then, though
the central text becomes fixed, the body of peripheral texts continues
to grow, as the central text is reinterpreted in new contexts.

In order to evolve, documents typically need to interact with the
world somehow.  The schedule of interactions could be arbitrary, but
when humans are involved some sort of phasic structure is often
involved too.  Laying down the law with some sort of schedule for work
on this document might be something I could make myself do (whether or
not keeping the schedule is also is not yet decided, and could be
interesting to document either way).  Personal schedules and
organizational schedules are related.  I really don't know what to do
(when things grind to a halt, but I assume it would be a good idea to
do something.
\end{notate}

\begin{notate}{Document as collection} \label{collection}
Maybe the evolution of the desktop that we hear so much about (or the
evolution of email, a la Gmail) is going to carry over to documents as
well.  Instead of being organized into sections, documents could just
be miscellaneous collections of notes, which then take on certain
organization through search (or maybe some fancy self-organizing
phenomena) as needed.
\end{notate}

\begin{notate}{Document as ontology} \label{ontology}
Documents typically come in an outline-oriented fashion.  The idea of
``document as ontology'' just expands on the idea of ``document as
container'' and posits some nested containers.  I'm not actually sure
what the difference between an ``ontology'' and a set of nested sets
is; my sense is that ontologies will typically also specify
interrelationships beyond just ``is-a''.  It would be worth looking
into this more.
\end{notate}

\begin{notate}{Document as simulasm} \label{simulasm}
Eventually when one has abstracted out the author a document takes on
the appearance of an agentless simulation.
\end{notate}

\begin{notate}{Unlimited refactoring}
The system seems to go one level beyond ``refactoring'' to something
more like ``multiple on-the-fly as-needed refactoring.''  Code will
take whatever shape you put it in temporarily; however, these shapes
can also be saved.  Maybe this is what refactoring is already about in
sophisticated systems (on a wiki it means something much weaker).
This would be worth investigating.
\end{notate}

\begin{notate}{Assimilating data} \label{assimilating-data}
We learn to map data ``up'' (local) hierarchies.  For example, when we
make lists of things to do, we map observations up from a perceptual
space to a potential action space.  It is valuable when living and
writing to have good heuristics for figuring out the best level for
observations.  (Some should be simple data, some should be containers
for other sorts of data.)  This also relates to the issue of text and
code from Note \ref{text-and-code}.

If the document is a process (Note \ref{process}), then it may
sometimes be a ``process of understanding'', or a trace thereof.  When
we think about something, we may be thinking in terms of an internal
``scholia-based document''.

As I've been working on \emph{this} document, various ideas have
occured to me, and they make their way in as marginalia, then as
Notes, and perhaps more expanded or refined.

It would be nice to have a specific reference to something from
``psychology'' here -- the \emph{On Intelligence} book would work,
even though it could be a better book.  Note that the Minsky book
contains similar (but more theoretical) notions.

The idea of ``hooking'' from Note \ref{thought-processes} seems
related -- both with regard to expressing assimilated knowledge by
supplying suitable hooks, and with regard to figuring things out by
hooking them together or hooking them to things that we already know.
\end{notate}

\begin{notate}{Text and code} \label{text-and-code}
What items should be coded up and what should be written about?  This
question seems particularly on-topic for a literate program.  (See
Note \ref{motto}.)  Historically, many thinkers were not able to
write (useful) programs, because there weren't computers to run the
programs on.  But algorithms have been being designed for a long time.
If Newton had kept calculus on a completely abstract and philosophical
level, it wouldn't have been useful for computation.  On the other
hand, I have very little idea how the phenomenologists could have gone
about coding up their ideas.  Even though now, work in knowledge
representation apparently follows in their footsteps.  Part of the
issue is that there are different languages for different things.
People frequently work on the ``conceptual'' level, and they need
things to be narrated to them.  Programs typically only ``work'', if
they come with some suitable narration, either intrinsic or extrinsic.
The actionable-nonactionable dichotomy may not be so useful (compare
Minsky on dumbbell theories).  The notion that things ``evolve'' from
nonactionable to actionable is probably a somewhat misleading, as
appealing as it might be.  Sometimes we need to ask: what programs
must be written to understand this question?  To paraphrase a
correspondent of mine (Ray Puzio), text and code are like yin and yang
(cf. ``chaos
linguistics''\footnote{{\tt http://www.thing.de/projekte/7:9\%23/chuang\_tzu\_linguistics.html}}?).
Understanding how they relate to one another in more detail would
presumably be quite fruitful.
\end{notate}

\begin{notate}{Hyperstuff} \label{hyperstuff}
Text is (or can be) hypertext; mathematics is just as ``hyperreal'' as
the HDM.  But I suppose that with some work towards \emph{formalizing}
a definition the ``hyper'' prefix might give us more than just hype
about a quote-un-quote new technology.  (As an example of how these
categories run together, Note
\ref{derivative-works-in-the-scholium-system} contrasts hypertext-like
indices with more text-like compilations.)

Accordingly, you can build an AI out of all sorts of miscellaneous
components.  Mathematics is already very AI-like, even though many of
its agents are human beings.  It isn't completely clear what the
differences there are between working with systems with more
electronic components and systems with less.  Is, or, how is, one set
inherently more powerful, or more risky, or just better for different
things?

Compare Note \ref{physical-scholium-system}.
\end{notate}

\begin{notate}{Not-so-holistic mathematics}
The QA section is typically split into two or three parts, each found
in a separate library.  Here in Minnesota, there is the math library,
the math philosophy section in the humanities library, and the
engineering library.  It can be somewhat inconvenient and a bit
offputting for people who want to look at math from all of these
perspectives.  On the other hand, it frequently makes sense to divide
when resources can't be copied.  Note that digital libraries don't
need to be divided in a permanent way; cf. Aaron's Emory paper -- in
both positive and negative senses -- in particular, think about
Google, today's great integrator.  Compare Leibnitz on the university
and Minsky on the mind.
\end{notate}

\begin{notate}{Philosophy of intelligent documents}
The main thing to look at here is the Dreyfus and Dreyfus article in
\emph{The Artifical Intelligence Debate}.  The ideas in the current
document (e.g. in Note \ref{human-and-artificial-intelligence})
should give some ideas about why there is hope for AI, and why we can
make something that people like to use as we make progress towards
this goal.
\end{notate}

\begin{notate}{Usefulness of bidirectional linking}
When a note contains a link to a section that contains more
information on the topic of that note, this gives the author a way to
have some of this topic spread into different contexts, but also to
have a compressed treatment that reflects all the different ideas
about the given topic in the document as a whole.  Bidirectional links
can be used to facilitate this bifurcated style of writing.
\end{notate}

\begin{notate}{Being ``meta''}
One way to be ``meta'' is to loop through some dimension that was
previously held static.  Just for example, we might switch from
single-article rendering to multi-article rendering at some point.
\end{notate}

\begin{notate}{Thought processes} \label{thought-processes}
It is interesting how different pieces of information will attach
themselves to a train of thought -- like so many carriages.  Or like
the miscellaneous stops on a messenger race.

In such races, there is actually no ``message'' to deliver; rather,
the racers must simply visit several predesignated stops, and
sometimes perform a certain task at each -- like drink a beer, for
instance.  Typically these stops can be visited in any order.

Like \emph{post facto} routes, thought process seems to be exploring a
large web-like space -- and only finally getting pulled together into
one narrative thread.  Perhaps one unwinds an \emph{Ariadne's thread}
as one explores; and perhaps at each important stop a hook is added,
so that when we get to the ``end'' of the path, we pull the Ariadne's
thread, and the hooked-in items from hither and yon are present before
us, ready in salute.

This business with the Ariadne's thread is pretty amazing, but the
hooks, by contrast, seem pointedly quotidian.  They include such drole
things as word-association, metaphor, alliteration, and various other
tropes.

Some thoughts are fleeting and remain between the lines forever (``you
should have seen the one that got away!'').  Is this a problem?
Perhaps less so when you are focused (Note \ref{staying-focused}).
\end{notate}

\begin{notate}{Mnemonics}
The fact that humans can often improve their ability to remember
things maybe means that we shouldn't trivialize memory-hooks
too much.  Ultimately, however, it seems like we do run into
``wetware limitations''.
\end{notate}

\begin{notate}{Search}
Being able to search is a very nice feature of the computer
interface.  But of course, it is also possible to use
thought processes (Note \ref{thought-processes}) to search.
(If it wasn't, we'd almost surely be utterly screwed.)
\end{notate}

\begin{notate}{Experience}
Experiences give you things to point at or hook to.  They are sort of
like pre-computed cases -- you can save time and be secure by knowing
what to do in a given situation in advance.  However, our experiences
are almost always (or maybe ``always'') only heuristically applicable.

Scholia, too, are sort of like pre-computed cases; i.e., they document
the experiences of persons who have interacted with a given subject.
\end{notate}

\begin{notate}{Establishing setting}
In filmmaking (and fiction writing), there is a tradition of images
that are only secondarily part of the story per se, but which, rather,
tend to say \emph{where} the story takes place, and to lend it some
realism (or similar quality of substance).

These images are sort of like attached scholia; they could be cut and
the plot would remain.  However, we often see films that have been
``ruined'' by overzealous cutting.
\end{notate}

\begin{notate}{This document as an example of CBPP}
This document will hopefully be CBPP (``for real'', and not just via
the Swift-Minsky societies of my mind).  The transition from solo work
to group work is probably almost always rocky, and sometimes even too
off-putting for it to happen at all.  However, there are certainly
some steps that can be taken to ease the process along -- it is
possible that having a ``cool'' product helps (some would argue
vehemently that the cool factor is unimportant to the extreme).

Hopefully by trying to make the code and documentation as accessible,
starting with some discussions about the system, moving to discussions
within the system, and from there soliciting code, etc., things will
work out for the current current project.
\end{notate}

\begin{notate}{Freedom, bookmaking, and book-buying}
In my view, part of the idea of paying for copies \emph{should} be
that the copy itself is a work of art.  If the copy is simply
something you could run off yourself on your laser printer, it might
not really be worth buying.  If it is a limited edition artwork -- and
if the price is right -- then it could be very much worth buying.  On
the other hand, if the salient features can be simulated cheaply, then
even the ``unique'' artwork may not be very valuable.

This relates to Section \ref{more-physical-scholia}.  The value of
print versions of a document is an interesting psychological question.
Writing marginal notes into a printed document seems to be a perfectly
viable way of building meaning clusters, but it isn't clear whether it
has any specific psychological benefits over writing marginal notes
into a sufficiently robust electronic system.
\end{notate}

\begin{notate}{Staying focused} \label{staying-focused}
In a scholia-based document, it should be somewhat obvious when you
drift off subject or off task.  Sometimes the connection between the
``off subject'' elements of the document and the ``on subject''
elements comes through a ``meta'' sort of connection.  For example, I
initially worte this Note as a reminder to myself to stay focused on
the code I'm looking at today, and not to get distracted by too many
peripheral or minor concerns.  Even though this Note is peripheral
with regard to the semantic content of the code, hopefully it will have
a salutory ``meta-level'' effect on the coding process.
\end{notate}

\begin{notate}{Hetrarchy} \label{hetrarchy}
If it isn't obvious, a hetrarchy is an example of a scholium-based
document in which the articles are people and the attachment
relationships are given by who is working on whose project.  It
shouldn't be too hard to map out these sorts of relationships in a
given CBPP system (like PlanetMath, for example).
\end{notate}

\begin{notate}{Hierarchy} \label{hierarchy}
Of course, a hetrarchy (Note \ref{hetrarchy}) can be viewed as the
joining-together of several hierarchies.  It seems helpful to be able
to look at the local hierarchies associated with certain articles.
Who cites who, and who do \emph{they} cite? -- and so on.
\end{notate}

\begin{notate}{Scholia-based digital libraries}
Some of the relationships to Digital Libraries are as follows.

In a physical library, marginalia are considered to be graffiti.  In a
digital library, they can easily be ignored, if that is desired.  They
are more like sticky notes than they are like true marginalia, but
even more potentiallly-ignorable than these handy-dandy things.

Still, the fact of the matter is that adding scholia to a document
will always change the document's meaning in some sense.  However, in
the case of \emph{private} scholia, other people will be in the dark
as to the nature of the change.  There is a gradient here; some
scholia. may be ``so public'' that everyone sees them, no matter how
their filters are set. (E.g., the user interface to monolithic
operating systems.)

In a physical library, loud conversation is considered to be
disruptive.  In a scholia-based document, it should be possible to
mark conversations with metadata that will make them more or less
private.  (Conversations are an example of processes-as-scholia.)  In
a typical feedback-forum situation, user feedback isn't actionable.
But of course scholia can be made actionable -- and they need not be
favorable.  For example, scholia can be used to vote such-and-such an
item out of the collection.
\end{notate}

\begin{notate}{The system} \label{the-system}
``The system'' is comprised exactly of the most-public articles.
Every user ``sees'' the system (as filtered through their own
customizations).  Freedom of users to make (not-publically-visible)
customizations \emph{ad infinitum} is certainly important.  But 
questions as to who can change the system itself and
\emph{how} it can be changed seem to be at least as important.
\end{notate}

\begin{notate}{Ontology versus pre-ontology}
Most of the work done so far in this document has been at the 
level of pre-ontology; work on the associate ontology or
ontologies can begin before long.
\end{notate}

\begin{notate}{Various free and nonfree cultures}
I still don't know what to say about the license-defined
common versus a managed public domain common commons.
\end{notate}

\begin{notate}{Teamwork}
Building a good team relates to the platform's usability (in both
directions).  Is the whole just as good as the sum of its parts?
What's good/bad/indifferent about \emph{informal} teams?
\end{notate}

\begin{notate}{Places to store thoughts}
Places, or spaces, for thoughts can be helpful to have around.  File
$X$ under ``experiments'', $Y$ under ``community spirit'' and so on.
As to whether the metaphor of ``file'' works, that's a good question;
labels may be uniformly better to use.  Furthermore, just about any
sort of attachment relationship can be used to categorize things;
compare Note \ref{hyperstuff}.
\end{notate}

\begin{notate}{Work ethic}
The achievement of a workable work ethic was supposed to be the big
event in Penmaster's Apprentice issue 2.  With regard to this work, it
is interesting to consider the interplay of different work styles:
small and large, physical and computational.  We can get very
different sorts of meaning clusters simply by adjusting our
scholia-creation patterns (at least in theory).  The (judicious) use
of summaries, critiques, new-found relationships, and so forth, give
an overall feeling to the evolution of a document.
\end{notate}

\begin{notate}{Interpretation}
What is meaningful?  In relationships, in activities, and so forth?
In literature, you want to figure out e.g. how to do character
development in a ``meaningful'' way, to create something that people
will want to read (and which they will find meaningful).
\end{notate}

\begin{notate}{Vanity publishing}
One's vanity can at least have the effect of encouraging one to write
and perhaps to do research as well.
\end{notate}

\begin{notate}{Role of philosophy in this document}
Some people have told me that they don't see the point of writing so
much.  But code is allegedly supposed to come after design, and design
is supposed to come after you understand the problem.  I think on
paper, and code things up as I understand them.  By this token,
perhaps eventually we'll see a version of the document in which most
of the philosophy has code associated with it.

One thing to realize (might as well do so at some point) is that this
is more than just a program; it is a philosophy of documents, whence,
maybe even a tentative metaphysics, since what can't be regarded as a
document, after all!, and as making some addition to the philosophy of
(the society of) mind.
\end{notate}

\begin{notate}{Philosophy of dictionaries}
Someone or other said something about defining via every \emph{use}.
Which implies that each definition is a more-than-octopodal sort of
thing, reaching all over the place.  Uses can often be partitioned,
and showing this partition only can massively simplify things.  It is
interesting to think about the scholium nature of partitions and
representatives of the partitional classes.
\end{notate}

\begin{notate}{Control}
One tends to imagine that the things that one can control are
relatively safe.
\end{notate}

\subsection{Mathematics of the scholium system} \label{mathematics}

\begin{notate}{Document topology}
The conversation about identification of document regions has a very
topological flavor, as does the idea (also mentioned in the paper)
about stitching documents together out of articles attached along
their margins.
\end{notate}

\begin{notate}{Cyberspace geometry}
The ``observation'' mentioned in Note \ref{observation-on-cyberspace}
could presumably be translated -- at least partially -- into a
statement about the hyperbolic geometry of cyberspace.  And I think
people have been studying this stuff for a while.  However, I think
there is more to the statement than just that.
\end{notate}

\begin{notate}{Region arithmetic}
The notion that regions can be copied, broken apart, deleted, and
added on the fly gives a notion of $*$, $/$, $-$, and $+$ for marked
up regions (and this applies to both text and markup).  I don't know
whether much can be made of these things, but they are the fundamental
operations, so perhaps there is something here.
\end{notate}

\begin{notate}{Clusion algebra}
Studying articles and clusions could be an interesting computational
variant on the study of objects and morphisms.
\end{notate}

\begin{notate}{Transclusion linearity}
The fact that transclusions shouldn't be indicated by constant, but
rather, linear, functions, seems cute (but I'm not sure what to make
of it).
\end{notate}

\begin{notate}{Scholium mathematics in general}
I'm particularly reminded of topology (including graph theory) and
ergodic theory.  ``Information theory'' seems related to the issues of
updates and versioning; similarly, ``knowledge theory'' may be related
to the issues of multi-directional updating.  From there we could move
on to game theory, I suppose, as the mathematical domain most
closely related to culture building (\emph{nomics}
are what I'm thinking of here; these also relate to MUDs).

Economics, more generally, seems to be where things are headed, when
we think about the social welfare situation associated with CBPP
system policies (e.g. rejecting corrections on PlanetMath has no
%% Maybe the open source version would be Franklinstein
repurcussions; if an essay about \emph{Frankenstein} is posted to the
site and the author rejects all corrections which say ``this is not
math!'', the administrators will presumably have to get involved, and
take some extra-protocol action to remove the article).

Other topics that should be taken up are the linguistic and
``mathematical artificial intelligence'' threads coming from the HDM
project.
\end{notate}

\begin{notate}{Network navigation}
If a scholium-based document is broken up into trees, the document's
underlying network can be linearized simply by ordering the roots of
these trees.  This relates to questions about literate programming and
proof checking (how do you write a ``structured proof''?).
\end{notate}

\subsection{Document analysis and the scholium system} \label{document-analysis}

\begin{notate}{Pitfalls of summarization}
It isn't completely clear that Note \ref{the-philosophy} makes a great
summary of the whole of the philosophy of this work.  Any summary
leaves things out or is otherwise vague.
\end{notate}

\begin{notate}{Recapitulation}
An article $A$ that appears after some article $B$ in a given ordering
which contains a summary of $B$.  This relationship could be indicated
by a link in one direction or another (likely to run from $A$ to $B$).
\end{notate}

\begin{notate}{Discourse analysis and holographic or monadological structures}
Long-range reference and design theory seem related to each other and
to this work as a whole; cf. Section \ref{mathematics}.
\end{notate}

\begin{notate}{Unified semantics}
Nelson (I think) has advertised some kind of unified theory of
semantics; it might be fun to take a look at that.
\end{notate}

\section{Conclusions} \label{conclusions}

\begin{notate}{Original conclusion}
In a context with multiple authors talking about many
intersubjectively important concepts, with ownership of the texts that
they write but freedom with the content, treating the total document
as a collection of scholia could provide important alternative
perspectives on a given subject.  The presence of on-screen scholia
could encourage responsible maintainership and aid users in making
specific comments and suggestions part of their critical reviews of
existing entries.
\end{notate}

\begin{notate}{Difficulty of putting things together}
Technological difficulties are posed by the fact that a document must
be assembled out of several pieces for presentation purposes and the
fact that standard web browsers and editors may not be very useful for
browsing or creating these sorts of documents.  (This isn't
\emph{always} a difficulty; rather, it motivates the current
development effort!)  Compare the message on the outside of the
Walker (Figure \ref{author-as-scholium}).

And: why didn't we see something like this document before?  Why
haven't the systems on the Hyperworlds server become ``popular''? I'm
not completely sure; there are probably a number of interrelated
reasons.
\end{notate}

\begin{notate}{Freedom in an online context}
Another very important thing to talk about is free versus non-free web
services.  I'm not sure if we mean to completely \emph{skewer}
non-free systems, i.e., systems that don't have a collaborative free
scholia-based compenent to them.  But on the other hand, such systems
really may be much less useful.  The current model is basically: if
you want to create some commentary on something else, you need to find
your own platform/soapbox from which to make that commentary.  Some
systems, like PM's attached fora, invite commentary and provide a
place for it.  But many, many, pages out there do not invite
commentary or simply do not supply a place to put it.  Another thing
to notice about the scholia-based system is that it is very
egalitarian.  Just because something was ``there first'' doesn't make
it better.  The commentary can become just as important as the main
document, or more important.  On the other hand, it would be very
useful to develop some algorithms that can be used to sort out the
useful commentary from the utter crap.  Slashdot does a pretty good
job of this, but I think there is still room for improvement.  And,
even though PM provides attached fora, they could do a significantly
better job of exposing the content that is provided in those fora in
the ``default view'' or some ``secondary view'' of the object that is
being commented on.  It might just take something simple, like
expanding all of the threads, or possibly using some Slashdot-like
rating system to expand notable threads.  Both interactive and
automatic ways of determining the usefulness of certain feedback can
be used -- and there are already some things like this in Gnus (I'm
sure).
\end{notate}

\begin{notate}{Knowledge communities}
It would also be good to talk about Ray's comments about knowledge
communities -- need to take another look at those again.  Also,
referring to other Ray stuff, I wonder what conditions would be
necessary for academics to accept more off-the-beaten-track
publications (e.g. PlanetMath).  Of course, if these things became
popular they wouldn't be off-the-beaten-track any more, really, but
let's not get circular.  My sense is that making some sort of medium
``respectable'' has to do with the social contract thing.  On the
other hand, people do seem to get respect for out-there stuff if it is
somehow novel, or useful, or \emph{something}.  (Not sure what the
relevant quality here is, how much ``uptake'' has to do with it, how
much ``propaganda'' (or ``PR'') or what -- if I knew maybe I'd try to
exploit this knowledge!)
\end{notate}

\begin{notate}{Social simplicity}
Sometimes complexity may not be totally avoidable within the code.
(E.g. if there are 1 billion scholia about a given article, sorting
them would take a long time.)  Sometimes we can do clever things --
but other times we can probably rely on the user groups to structure
things in ways that won't push things too far.  It isn't clear how
much we can rely on them for this -- but my guess is that things that
the machine would whine about are frequently going things that people
want to avoid for their own reasons.
\end{notate}

\begin{notate}{Human and artificial intelligence} \label{human-and-artificial-intelligence}
The HDM project may be better off building something that smart people
can hack than building smart code right away.  Perhaps that's why I'm
giving precedence to the scholium system over things like parsers and
the like.  The ``human element'' in software development seems
powerful.  It is curious that I've envisioned the scholium system as
being mainly about the AI component of HDM; but it may be that we can
expect more from it, namely, a contribution to AI for HDM creation.
Take a look at Note \ref{text-and-code} for some words against
thinking about things in terms of ``evolution towards actionable
form.''  I think a better argument would be for \emph{co-evolution}
(which is good, because there has been plenty of thinking done on this
issue).  SNePs seems to have some things to say on this
topic\footnote{{\tt
    http://www.cse.buffalo.edu/sneps/snerg.prospectus/node19.html}}.
Presumably so do others; still, by contrast with many other systems,
the co-evolution view seems to be emphasized particularly strongly
here.
\end{notate}

\begin{notate}{Labels vs namespaces, transclusion vs inclusion, overlays vs text properties}
It is interesting to compare the ``hard'' to the ``soft'' approaches,
both of which appear in this document in different roles (sometimes in
interestingly similar relationships to one another).
\end{notate}

\begin{notate}{Compatibility between versions of this system?}
All else equal, it would be nice for any future versions of this
system to be compatible with this one.  But we can make no guarantees
that this will actually happen, although I do think that the system
will be revised somewhat to continue to make improvements and add new
features.
\end{notate}

\begin{notate}{Future work}
It would be pleasant to have Arxana hooked up with a graphical
node/edge display, such as fenfire\footnote{{\tt http://fenfire.org}}.
Building rich displays is a problem that we'll want to consider from
many angles (cf. Note \ref{large-form}).  The usefulness of getting
various tools worked out in plain-text modes shouldn't be ignored!
\end{notate}

\begin{notate}{Getting work done on Arxana}
We mentioned one strategic point, that of collaborating early on, in
Note \ref{collaborate-early}.  There are probably other things I
should be doing that would enhance my ability to get things done with
this system.  Productivity is a topic that should be studied more.
Working a lot, trying to check concrete usefulness or applicability a
lot, maintaining a strategic plan -- these all seem like good ideas.
\end{notate}

\appendix

\section{Appendix: A simple literate programming system} \label{appendix-lit}

\begin{notate}{The literate programming system used in this paper}
The point of this code is to grab all the Lisp sections
of this document and evaluate them.  It requires that
the \LaTeX\ be written in a certain consistent way, with
all of the code in a certain kind of environment.  That
is all it does and all it requires.

\begin{verbatim}
(defvar lit-code-beginning-regexp "^\\\\begin{lisp}")
(defvar lit-code-end-regexp "^\\\\end{lisp}")
(defvar lit-count 0)

(defun lit-eval ()
  (interactive)
  (lit-process 'eval))

(defun lit-process (&optional code)
  (interactive)
  (setq lit-count (1+ lit-count))
  (save-excursion
    (let ((to-buffer (concat "*Lit Code " (int-to-string
                                            lit-count)"*"))
          (from-buffer (buffer-name (current-buffer))))
      (set-buffer (get-buffer-create to-buffer))
      (erase-buffer)
      (set-buffer (get-buffer-create from-buffer))
      (goto-char (point-min))
      (while (re-search-forward
              lit-code-beginning-regexp nil t)
        (let* ((beg (match-end 0))
               (end (save-excursion
                      (search-forward-regexp
                       lit-code-end-regexp nil t)
                      (match-beginning 0)))
               (match (buffer-substring beg end)))
          (save-excursion
            (set-buffer to-buffer)
            (insert match))))
      (case code
       ('eval
        (set-buffer to-buffer)
        (eval-buffer)
        (kill-buffer (current-buffer)))
       (t
        (switch-to-buffer to-buffer))))))

(defun lit-require (feature)
  (let* ((basename (symbol-name 'scholium-preliminaries))
         (name (concat basename ".tex"))
         (livep (buffer-live-p name)))
    (save-excursion
      (if livep
          (set-buffer name)
        (find-file name))
      (lit-eval)
      (unless livep
        (kill-buffer (current-buffer))))))
\end{verbatim}
\end{notate}

\begin{notate}{Another way}
This code could go into a stand-alone file.  This would
make it somewhat more reuseable, at the cost of making
the current document somewhat less self-contained.
Since I seem to like writing literate programs, making
the code reusable probably wins out.
\end{notate}

\begin{notate}{Adding docstrings at deliteration time}
It might be adventageous to have some way to add docstrings to
functions that have been literately documented when the literate
documentation is stripped out, before the functions are evalled.
\end{notate}

\begin{notate}{Split into sections}
It might be handy to have the code that evaluates
this file split it into several separate files, or
to provide an option for doing this in the ancillary
elisp file.
\end{notate}

\begin{notate}{LaTeX verbatim environments}
The only thing you can't put into a verbatim environment is the text
``\verb|\end{verbatim}|'' $\ddot{\smile}$.  See Note
\ref{principle-of-transparency}.
\end{notate}

\section{Appendix: Overview of specifications} \label{appendix-specs}

\begin{notate}{A quick guide for the theoretically minded}
This section will contain an overview of the specifications of
\emph{this} scholium system, which, I think should help people who
want to hack on it, or who want to create \emph{other} scholium
systems.  The full theory is found in the main document, of course;
this section serves to remind the reader of points that are dealt with
in greater detail there.
\end{notate}

\begin{notate}{Spec for article}
An article is an element of some namespace, by default,
the main article-table (a hash table).  
It takes the form of a quintuplet,
\begin{idea}
(name text about type bookkeeping)
\end{idea}
where, when stored in the hash table, \inp{name} is used
as the key.  (See Note \ref{structure-of-articles}.)
\end{notate}

\begin{notate}{Spec for about data}
About data in articles takes the form of a list of \emph{links}.  (See
Note \ref{format-of-about-field}.)
\end{notate}

\begin{notate}{Spec for link}
Each link takes the form
\begin{idea}
(<target article's name> &rest <link-types>)
\end{idea}
The design here maintains partial symmetry between the treatment of
article types and link types; the cdr of a link can be processed
by the same typedata processing functions as the \inp{type}
data from articles.  (See Note \ref{links}.)
\end{notate}

\begin{notate}{Spec for type data}
A type can be a single token specifying a type or a list of types.
Types can, themselves, come with additional data; in which case they
take the form of a list, where the car gives the type's name, and the
cdr carries all additional specifics.
\end{notate}

\begin{notate}{Spec for metadata article}
An article $A$ has metadata article $\tilde{A}$ whose
\inp{name} is 
\begin{idea}
(meta <name of A>)
\end{idea}
The \inp{text} of a metadata article takes the form of a list of
lists, with each sub-list headed by a tag naming the contents of that
sub-list.  The \inp{about} data is `nil', the \inp{type} is `meta',
and the bookkeeping data is `system'.  (See Note
\ref{metadata-article-design}.)
\end{notate}

\begin{notate}{Spec for backlink}
An article's backlinks are stored on the `backlink' field of its
metadata article.  An individual backlink
takes the form
\begin{idea}
(<name> <link number> [<version number>])
\end{idea}
where ``name'' is the name of the article being backlinked, and
link number indicates the particular link to which this backlink
corresponds.  The optional version number is used if the backlink
corresponds to a specific version of the linking article other
than the most current one.
(See Note \ref{implementation-of-backlinks}.)
\end{notate}

\begin{notate}{Spec for link-id}
The purpose of a link-id is to specify a link when marking up a piece
of text with a text property corresponding to a scholium, namely the
link through with that region was indicated (see Note
\ref{the-scholia-property}; the `scholia' text property contains a
list of link-ids.).

The format of a link-id is
\begin{idea}
(<name> <link number>)
\end{idea}

Notice that link-id's also come up in the context of masks and other
places.  They are a sort of idiom used in several places in the work
(as you can easily see by looking around at the specs shown in this
appendix).  One thing to notice is that the ``usable about data'' of
Note \ref{compute-usable-about-data} \emph{et seq.} leads up to markup
fairly directly, and, in particular, that it computes markup by
applying masks to links as needed.  So, it is no wonder that there is
some continuity between these various parts of the system; and, in
particular, the fact that link-id's are a part of that continuity
makes sense.
\end{notate}

\begin{notate}{Spec for mask} \label{spec-for-mask}
An article's masks are stored in the `masks' field of its metadata
article.  An individual mask takes the form
\begin{idea}
((<name> <link number>) &rest regions)
\end{idea}
The regions are two-element lists, each giving the beginning and end
of a region that the link is being redirected to.  See Note
\ref{masking-links} and Note \ref{masks}, and for further discussion
of the lack of generality in the current mask format, see Note
\ref{generality-of-masks}.
\end{notate}

\begin{notate}{Spec for `usable-about-data'}
Something called ``usable-about-data'' is produced by the function
`compute-usable-about-data' (see Note
\ref{compute-usable-about-data}).  This stuff is used when marking
things up (in particular, by `mark-up-scholium' and
`mark-up-reference').  The reason for documenting it here is that its
format is confusing and easy to forget.  I think that it would be best
if we could find another way to get this data to where it needs to be.

The format of the return value of `compute-usable-about-data'
is a list of elements of the form
\begin{idea}
((<name> <link number>) <beg end | nil>)
\end{idea}
where name is the name of a scholium, `beg' and `end', if specified,
give the beginning and end of a marked region (yes, only one region
per link, as mentioned in Note \ref{links-indicating-passages}),
whereas `nil' is specified if the link indicated by the `car' of this
form points at the whole of the article being rendered.  

(I am not a huge fan of this format, but it does seem to work;
however, maybe the name should be changed to ``usable about data for
markup'' or something like that, to make it clear why we are focusing
on data describing regions.)

(Also, perhaps we should note in this spec that instead of the
\inp{name}, sometimes ``(mask \inp{name})'' is used instead.  The
development here seems a bit confusing: if we use (mask \inp{name}),
is this a proper link-id?  Something to think about, I guess.)
\end{notate}

\begin{notate}{Comments on specs}
I fear that the choice to allow a token or a list of types may come
back to bite later; it is certainly somewhat annoying now.  But I
think we're running with it?

Eventually we might want some specs for various operations or other
objects (e.g. transclusions are both operations and static objects).
But we can hold off on that til it is all defined.

Furthermore, we'll want to have clear specs for the formats that are
used \emph{in markup} (of regular links, links coming from references,
fake references, possibly etc.) but we don't quite have that taken
care of yet.
\end{notate}

\section{Appendix: Time and workflow management} \label{appendix-cal}

\begin{notate}{Framework for keeping track of time and events}
People have been using calendars since the beginning of time
(literally!).  However, this is a case where reinventing the wheel
might be called for.  The scholium system can be used to keep track of
epochs of different scales, and maintain the proper relationships
between them.  Well, different calendar programs already exist that
will e.g. keep track of the events that are taking place in all of the
days of the month, and know (in some sense of the word) that these
events are taking place in that month.  But I think I can do somewhat
better for my needs.  One of the reasons is to keep a lot of data in
the system and integrate it up to get certain conclusions.
Furthermore, I want to keep track of the way big tasks get parceled
out to smaller time-scales.  Another useful feature is the ability to
go in the opposite direction, and produce somewhat realistic
predictions about what can be done by when.

So, we'll give it a shot.  

(Presumably if this ``appendix'' gets extremely verbose, big parts of
it should be chunked out into some other separate document.)
\end{notate}

\begin{notate}{Summer 2006}
I want to get a releasable version of Arxana out.  I have to help
mentor Shaneal on his proof checking stuff.  I'd like to get started
with some NLP stuff.  I'd want to keep up with general reading and
correspondence related to the HDM project.  And finally, I want to do
some more math related things.
\end{notate}

\begin{notate}{Miscellaneous items that can get into release} \label{misc-items-for-release}
This stuff has come up as potentially important or at least
interesting to consider for the release.
\begin{itemize}
\item How about being able to spawn a semi-anonymous scholium rooted
  in the current article?
\item Build a subdocument library.
\item Make an easy to use demo.
\item Get a system for unique ID's working.
\item Import APM-Xi.
\item Go back over the internal formatting and display, and make sure
  that the format is actually used properly by the code that is
  supposed to use it.
\item Finish putting together the ``restricted view'' mode.
\item Perfect some system for managing clusions (including a
  simplified version for assembling a document from outlines).
\end{itemize}
\end{notate}

\begin{notate}{Revising the schedule}
One of the most reasonable-seeming things to do when starting work is
to assess the schedule of events, and clarify exactly which activities
will be done during that work session.  Other clarifications to the
schedule can be effected at this point too.
\end{notate}

\begin{notate}{Release conditions}
It would be nice to get some sort of system set up for collaborative
work on the same state-containing scholium system.  Maybe Arxana will
have to be made inherently peer-to-peer for this to work (given that
Emacs doesn't seem to include features for operating as a server with
several distinct users working on it at the same time).  Or
conceivably we can find a good back-end (e.g. in Common Lisp or
scheme?) to connect people to.  Another possibility that might work,
although it sounds like a little bit of a long-shot, is to get
Ebby\footnote{{\tt
    http://dev.technomancy.us/phil/browser/dotfiles/.emacs.d/ebby.el}}
working with Gobby, or build on this system to make some
Emacs(-friendly) multi-user environment.  (Since Ebby apparently just
works as a client, I currently suspect that it won't directly solve
our state-maintaining problems.)

Since this all looks tricky, it seems to me that having some
acceptible (though certainly not perfected, that would be asking too
much) single-user environment ready to go even before making too many
more investigations of the multi-user stuff.  See Note
\ref{misc-items-for-release} for the key details on this topic.
\end{notate}

\begin{notate}{Tue Jun  6 15:12:52 2006}
I write this section up to the current point.  Then I think about what
to do next.  I guess it would be a good idea to look at importing
APM-Xi again, and maybe try to get the subdocument library stuff
working.  Since this should be fairly easy (at least initially).
\end{notate}

\section{Appendix: Annotated non-bibliography} \label{appendix-bib}

\begin{notate}{Outside references}
For now, references are contained in footnotes.  To view a compilation
of the Notes that contain footnotes, evaluate this form:
\begin{idea}
(progn (article-menu-list-articles-matching-regexp "\\footnote")
  (listing-to-compilation nil))
\end{idea}
Eventually we may have a nicer way of presenting this data.  Also,
there are a number of references made in the text for which there are
no footnotes, and the footnotes that do exist don't necessarily take a
proper or in any way standard form.  This can be fixed up eventually.
\end{notate}

%%%%%%%%%%%%%%%%%%% Back matter %%%%%%%%%%%%%%%%%%%%%%

\clearpage

% \addcontentsline{toc}{section}{\numberline {}References}
% \bibliography{sbdm4cbpp}

\printindex

\end{document}

%%% sbdm4cbpp.tex ends here