Created Minutes for day two: 8 February 2017 (markdown)

[xproc-workshop-2017-wiki.git] / Minutes-for-day-one:-7-February-2017.md

# XProc Workshop Minutes, Day 1, 7 February 2017

## Background links:

-   Spec repo: <https://github.com/xproc/1.1-specification>
-   Online specs: <http://spec.xproc.org/>
-   Issues list: <https://github.com/xproc/1.1-specification/issues>
-   Norm’s @depends proposal:
    -   <https://github.com/xproc/1.1-specification/pull/33>
    -   <http://xpspectest.nwalsh.com/depends/head/xproc11/>
-   Norm’s proposal for issue #30:
    -   <https://github.com/xproc/1.1-specification/pull/32>
    -   <http://xpspectest.nwalsh.com/fix-30/head/xproc11/>


<a id="orgee13667"></a>

## Agenda / Minutes


<a id="org88438ac"></a>

### Introductions

-   Present: Norm, Romain, Achim, Christophe, Matthieu, David, Geert,
    Gerrit, Martin, Henry, Liam [afternoon]


<a id="org08ee05f"></a>

### New Action Items

-   Norm to ask Nic for feedback about the state of the community group
-   Norm to write a message (periodically?) to xproc-dev explaining
    that it’s being used both for user and spec-dev discussions.
-   Norm to send Romain a couple of example tests to try converting
    to XProcSpec.
-   Norm to write better documentation of the test suite format
    (and fix the schema)
-   Henry to write a straw man for the non-XML documents question


<a id="orgcb61fce"></a>

### Review of the agenda

-   Review of the test suite mechanics
-   What’s the status of the W3C community group?
    -   Do we need to make it more present?
-   What version number do we use? What is “V.next”?
    -   Are we going to call it “XProc”?
-   Development infrastructure
    -   GitHub? Wiki’s provide no notifications
-   What are the important workflows?
    -   Center of gravity of group is clearly publishing
    -   Revisit “pain points” from the bottom up
    -   Are database (relational or XML) interfaces important?
-   What’s our thinking on the “resource manager”?


<a id="orgcf8688b"></a>

### Brief review of the previous workshop minutes

-   What about backwards compatiblity?
    -   A goal, not a requirement; should be able to write a tool
        to transform old pipelines to the new spec
    -   Do 1.1 pipelines ever have to run 1.0 steps?
-   Open question remains: are we going to try to do a “1.1” quickly
    that’s a small(ish) delta on 1.0 or are we doing something grander?


<a id="org02f97a6"></a>

### What is it called, what version is it?

-   Norm mutters on about the options a bit. 1.1, 2.0, 3.0?
-   Martin: I’d like a 2.0 version to better engage folks who 
    aren’t currently using it.
-   Achim: Then maybe we should call it 3.0?
-   Geert: I think 3.0 is a real possibility. Leaves behind the
    legacy.
-   Norm: We could also change the name.
-   Henry: No, we should keep it. It doesn’t have to carry baggage.
-   Achim: I think 3.1 makes sense; it’s using XPath 3.1 and XDM 3.1
-   Matthieu: I don’t think these have to be this closely aligned.
    What about other versions of XPath in things like Schematron.
-   Christophe: If we go to 3.x, we need to bring something new.
    If we only optimize 1.x, it won’t be “new enough”.
-   Matthieu: Wouldn’t the same be true for 2.0?
-   Henry: We have to come back to the question of what it is we’re
    naming.
-   Christophe: It’s not clear what 1.1/2.0 is. But maybe 3.0 could
    bring clarity.


<a id="org953a61e"></a>

### What is it we’re building then?

-   Norm gives a potted summary of the “1.1 will be a better 1.0”;
    no, “we need a totally new 2.0”. No, the community wants a better
    1.0 (at least an XML syntax). So here we are. What do all y’all
    want?
-   Romain: I’d like two things: a better XProc 1.0, trying to fix
    the usability shortcomings. And the other would be a more ambitious
    2.0 with a compact syntax, etc.
-   Norm: I think the only certainty is you can’t have both.
-   Romain: Pragmatically then an improved XProc 1.0.
-   Achim: I agree; the compact syntax in particular has technical
    problems.
-   Henry: None of us are programming language designers; most design
    fails. There’d be real benefit to expertise in that space. We
    should stay with what we’re good at.
-   Achim: So a better XProc 1.0, is that the consensus?
-   Matthieu: I think there are things that we can do to make it
    easier to use that go a little bit beyond “an improved 1.0”.
-   Achim: The current spec drops the XML centric view.
-   Henry: There’s a spectrum between “a data flow language” on one
    hand and “a web-centric language with a first-class place for
    non-XML formats”.
-   Geert: Do we know what we want with non-XML documents? Are we
    building database migration tools? I think we all want the same:
    an occasional CSV, images, zips. As publishers, we want to work
    with XML documents. I don’t need the extreme spectrum of non-XML
    documents, it’ll be an XML workflow with the occasional non-XML
    document.
-   Henry: I think the crucial first step is to say that “here’s a JPG,
    I can route it, I can package it,” but I don’t need to have
    steps that work with it.
-   Norm: I think JSON documents are the elephant in the room.
-   Martin: Do we agree that if we want to use JSON, we have to
    represent them as XML internally?
-   Norm: Well. Maybe. That’ll always be possible, but other users
    might want more.
-   Henry: Documents have media types. But that’s not a complete answer.
    Is it just what you got off the wire, is it a serialization, or
    does it move XDM data structures around. The question then is:
    do we support anything other than documents with a media type
    and XDM instances?

Some discussion of the steps that would/could/might implement
operations on random documents.

-   Achim: What about the mapping to XDM because we use XPath
    as the expression language.
-   Norm: I think you just have decide what it means.
-   Henry: I think the minimum needed to declare victory is to
    give implementations some freedom leaving “error” as a reasonable
    alternative. We don’t need to provide *in the spec* the extension
    story.

-   Henry: There’s a question of what flows through the pipeline:
    is it just sequences of XDMs? Right now we have both: a port
    can declare if it accepts a sequence or not. We could get
    rid of that in two ways: by saying everyone has to accept
    sequences or by saying that the compilation step will take
    care of that: if a sequence arrives, a step will be invoked
    multiple times.


<a id="org5170dce"></a>

#### Properties of what we’re building

-   It has an XML syntax reminiscent of the XProc 1.0 syntax
-   It has a 3.1 data model
-   It uses XPath 3.x as the expression language
-   Variables/options are now allowed to be any data model instance
-   No parameter ports
-   Metadata flows through the pipes
-   Documents in the pipes have media types
-   Attribute value templates (AVT) and
    text value templates (TVT) are supported
-   Ability to import user-defined function (XSLT or XQuery)
-   New if/then/else compound step?
-   Variables everywhere
-   Dynamic evaluation of pipelines

##### Step improvements

-   SVRL output for all validation steps (a report port)
-   XML-centric step vocabulary
-   Move the eXProc.org steps into a standard step document


<a id="orgeec1887"></a>

### What is it called, what version is it? (Continued)

-   Norm: Now that we have more clarity on what we’re building,
    do we know what to call it?
-   Henry: I think “XProc 3.0” is good. If we ever do the
    research project, we call it something else.

General murmers of agreement around the table.

-   Norm: “XProc 3.0” is proposed; any objections?

The “2.0” version has been burned; it’s too difficult to explain
what it means. “3.0” is a clean break and will use the XPath 3.x
expression language.


<a id="orgdb3e29d"></a>

### What are the *semantics* of pipelines?

-   What are the lowest-level abstractions needed to
    describe/discuss pipelines?

-   Norm waffles on a bit; his simplification is just
    steps with connections between them.

Some discussion of variables and ordering of steps. There’s
clearly a question of expressing the order in a pipeline syntax
that’s not quite the same as what the underlying implementation
has to do.

-   Henry: The draft I have (which I’ve never published) retains
    from XProc 1.0 the idea of a context that has a bunch of
    bindings. It also has the idea that setting a variable is
    a little step. Then I realized that that set variables off to
    the side of the flow of documents through the pipeline between
    “real” steps. I was worried about pushing/popping context
    but maybe that’s not the right question; that’s an artifact
    of the surface syntax.
    … The other aspect of complexity in the system I was developing
    was probably not necessary for what we’re calling 3.0. It attempted
    to distinguish between steps/pipes and sources/syncs.
    … If you have something that steps outside the dependency flow,
    how do you manage the synchronization issues (for example,
    updates to a database) that arise. That something is a resource
    manager.
-   Norm: I’d not considered the question of synchronizing something
    like database access across several steps.
-   Henry: The Markup pipeline had many extra steps to compile
    schemas and stylesheets that it put in the resource manager.
    The steps that implemented schema validation and XSLT always
    expected a URI (discharged by the resource manager) to their
    stylesheets and schemas. That imposed an extrinsic order.
-   Henry: Why a resource manager? It was a simplification: it was
    easier to think about producing an output document and giving it
    a name and then later using that name. It was an alternative
    to pipes.
-   Norm: Use pipes if you can, but a resource manager does handle
    the case of wanting to generate a document that will be
    XIncluded (for example).
-   Henry: You could also imagine that you have code that produces
    a RELAX NG document and your Relax engine doesn’t have a standard
    input port, it expects a URI.
    … The other use for a resource manager was for internal use.
    Some notion of preprocessing or compilation that you want to
    use.


<a id="org409905c"></a>

### Development infrastructure

-   Norm: I think it’s important to do the work in the open.
    Github is certainly convenient in terms of tooling. The
    wiki notification problem is real, but are there other
    difficulties?
-   David: What do we need?
    -   A mailing list (we have one)
    -   Github is good for code and specs
    -   Documentation wiki is just not very good for docs
-   Achim: Do we have a mailing list? The xproc-dev list
    isn’t very good for 3.0 discussions because many folks
    don’t understand.
-   David: If we discuss in email then maybe the don’t need
    notifications.
-   Norm: The other possibility is to just commit documents
    to a repo; we can get notifications for that.
-   Geert: We need to make sure we aren’t overwhelming the
    “occasional users” on the xproc-dev mailing list.
    … There will be other work on tests, tutorials and other
    documents that we can make to help 3.0 users.
-   Norm: I think it would be great if other community members
    could take on maintaining the tutorials and things. It’s
    very hard to do both the spec and tutorials.

Some discussion of writing a primer or tutorial. Norm observes
that if someone writes one, we can check it into the repo and
it’ll be published automatically.

Some discussion of existing resources: the spec, tutorials, etc.

-   Christophe: I volunteer to work on some tutorial material.
-   Matthieu: I will work on that as well.

(Thunderous applause.)

-   Achim: The other point is test cases. We need new tests.


<a id="org06fd04f"></a>

### Test suite/running tests

-   Norm summarizes the current state of tests
-   Romain: There’s also an XSpec test.
-   Achim: There’s also the problem of multiple outputs.
    … If, for example, you don’t support PSVI, then the output
    of the test should be “an error” if you don’t specify PSVI
    and a document if you do. Both of those should qualify as
    a “pass”.
-   Norm: We could probably extend the test suite to cover that.


<a id="org6bd3e15"></a>

### Review of the current working drafts

-   Current state of the specs

-   Achim leads discussion of
    <https://github.com/xproc/Workshop-2017-02/wiki/Some-comments-on-XProc-1.1>-(2017-01-22)

-   Discussion highlights:
    
    -   What do we do about the p:document-properties() function; if the document
        is an argument to the function, what XDM value does it have when the
        document is not XML?
    -   Some discussion of the impact of sequences on this function
    -   Achim: What is the content-type of a sequence of three integers?
    -   Henry: Maybe only documents have a media type.
    -   Norm: “(1,2,3”) content-type=“x.vnd/xdm-item”
    -   Henry: We have said that arbitrary documents flow through the
        pipeline; but variables can hold arbitrary XDM types
    -   Achim: But only documents have metadata
    -   Henry: Variables can contain any XDM item. Pipelines
        consist of sequences (possibly unitary) of documents with
        a media type and associated metadata.
    
    Further discussion of what limits we can/must/should put on
    the kinds of things that can flow through a pipeline.
    What happens to an XQuery that produces three numbers.
    
    Achim outlines the ideas from his comments document about mapping
    from arbitrary documents to XDM instances.
    
    Henry: I’d do it by having the output be a sequence of documents
    plus metadata. What we get is the media type and maybe a URI. (Or
    the filename in the case of a package.) Maybe you have to run a
    step to do the “file” magic to add a media type if it was absent.
    Now I can write a choose, based on the media type, that is inside
    a for-each. I can use document-properties() for that single
    document.
    
    Achim: I think we can put the properties in the context so that
    we don’t have to load the document.
    
    Henry: You’re absolutely right on two counts: it has to not be
    an error and you have to be able to get at the properties.
    
    Henry and Romain discuss the propsect of using a handle and
    a special load function to get the data.
    
    David: I’m concerned about using MIME types. What about encoding
    and MIME type parameters?
    
    Achim: Mapping from text/plain to string isn’t quite right.
    But there are only four types.

– BREAK —

Achim: Two questions: what flows on ports and what happens if
that becomes a context node. Why not restict whatever arrives
on the port to XDM? The problem is with the current, abstract
notion of documents in the spec.

Henry: We could just us an XProc private namespace to wrap
a handle for everything else.

Achim: What happens if some non-XML representation appears as
a context node. The spec says that this is implementation
defined or an error. I tried to rescue this by having a
mapping just for the case where the non-XML document node
is used as a context element.

David: We’re not talking about mime types, we don’t mean
mime types. We’re talking about types. We can map from
media types to our types, but we should have our own types.

Liam: I don’t really think that’s a good idea. This is what
mime types are for. You could make parameters available as
metadata.

Norm: If we were going to go the handles route:

1.  An XML document would be … an XML document

1b. An JSON document would be … a JSON XDM representation
    (an array or a map)
1c. XDM items other than documents … ???

1.  A text/\* document would be
    <p:handle type=”text/plain” handle="12345">…the text…</p:handle>
2.  Any other document would be
    <p:handle type=”media/type” handle="5678"/>

The argument to the p:document-properties function is a node
and it has to be a valid node or handle.

Achim: This is the solution I like least. We can do things
which are not XML. When people ask, we show them XML documents.

Henry: No, this is just a way of having blobs that flow
through the pipeline.

Henry: Step implementations never see the handle markup.

Norm: This abstraction is too leaky.

Henry: There’s basically a difference of view here. I’d like to
deal with the general case in the general way and look at special
cases for text later. Others are trying to deal with the special
cases but don’t care so much about the general case.

Achim: What bothers me is whatever is a representation, what
happens if it becomes the context node.

Geert: I’ve been wondering if the way NetKernel solves some
issues is relevant. They have a URI for every object and these
URIs have types. There’s a “transreption” when necessary to use
the representation. Most of this is an implementation detail.
If you want to use it, for example for XSLT, they define a
mapping from each representation to something appropriate
for XSLT.

Norm: Yes, I think a layer for doing transformations of
representations is a good idea.

Geert: One of the things that makes it work is that
the transreption library is small and fixed.

Romain: I think we are missing key use cases that would help
us make this decision.

Achim: We could compare the solution with some different use
cases. We have the one that I defined.

David: I’m thinking about cases where I create RDF and I want
to serialize it as ntriples.

Liam: Going the other way would give you non-XML: loading
some ntriples into an RDF graph.

Norm: zip manipulation, RDF graphs, JSON objects, text documents
images, CSVs.

END OF DAY ONE