Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1452272 > unrolled thread

[ANNOUNCE] git-series: track changes to a patch series over time

Back to article view | Back to linux.kernel

Contents

  [ANNOUNCE] git-series: track changes to a patch series over time Josh Triplett <josh@joshtriplett.org> - 2016-07-29 08:50 +0200
    Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-07-29 12:20 +0200
      Re: [ANNOUNCE] git-series: track changes to a patch series over time Josh Triplett <josh@joshtriplett.org> - 2016-07-29 13:10 +0200
        Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-07-29 14:50 +0200
          Re: [ANNOUNCE] git-series: track changes to a patch series over time Josh Triplett <josh@joshtriplett.org> - 2016-07-29 15:10 +0200
            Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-07-31 16:40 +0200
          Re: [ANNOUNCE] git-series: track changes to a patch series over time Stefan Beller <sbeller@google.com> - 2016-07-29 19:00 +0200
            Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-07-31 16:20 +0200
      Re: [ANNOUNCE] git-series: track changes to a patch series over time Christian Couder <christian.couder@gmail.com> - 2016-08-01 07:10 +0200
        Re: [ANNOUNCE] git-series: track changes to a patch series over time Eric Wong <e@80x24.org> - 2016-08-01 10:10 +0200
          Re: [ANNOUNCE] git-series: track changes to a patch series over time Josh Triplett <josh@joshtriplett.org> - 2016-08-01 11:00 +0200
            Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-08-01 12:00 +0200
            Re: [ANNOUNCE] git-series: track changes to a patch series over time Eric Wong <e@80x24.org> - 2016-08-01 23:30 +0200
    Re: [ANNOUNCE] git-series: track changes to a patch series over time Stephen Warren <swarren@wwwdotorg.org> - 2016-08-01 17:30 +0200
      Re: [ANNOUNCE] git-series: track changes to a patch series over time Josh Triplett <josh@joshtriplett.org> - 2016-08-01 21:50 +0200
    Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-08-03 21:30 +0200
      Re: [ANNOUNCE] git-series: track changes to a patch series over time Josh Triplett <josh@joshtriplett.org> - 2016-08-05 00:50 +0200
        Re: [ANNOUNCE] git-series: track changes to a patch series over time Richard Ipsum <richard.ipsum@codethink.co.uk> - 2016-08-10 20:40 +0200

#1452272 — [ANNOUNCE] git-series: track changes to a patch series over time

I'd like to announce a project I've been working on for a while:

git-series provides a tool for managing patch series with git, tracking
the "history of history". git series tracks changes to the patch series
over time, including rebases and other non-fast-forwarding changes. git
series also tracks a cover letter for the patch series, formats the
series for email, and prepares pull requests.

This makes it easier to collaborate on a patch series, distribution
package, backport, or any other development process that includes
rebasing or non-fast-forward development.

A patch series typically goes through multiple iterations before
submission; the path from idea to RFC to [PATCHv12 1/8] includes many
invocations of git rebase -i. However, while Git tracks and organizes
commits quite well, it doesn't actually track changes to a patch series
at all, outside of the ephemeral reflog. This makes it a challenge to
collaborate on a patch series, distribution package, backport, or any
other development process that includes rebasing or non-fast-forward
development.

Typically, tracking the evolution of a patch series over time involves
moving part of the version control outside of git. You can move the
patch series from git into quilt or a distribution package, and then
version the patch files with git, losing the power of git's tools. Or,
you can keep the patch series in git, and version it via multiple named
branches; however, names like feature-v2, feature-v3-typofix, and
feature-v8-rebased-4.6-alice-fix sound like filenames from corporate
email, not modern version control. And either way, git doesn't track
your cover letter at all.

git-series tracks both a patch series and its evolution within the same
git repository. git-series works entirely with existing git features,
allowing git to push and pull a series to any git repository along with
other branches and tags. Each time you change the patch series, whether
fast-forwarding or not, you can "git series commit" a new version of the
patch series, complete with commit message.

You can rebase a patch series with "git series rebase -i", format it for
submission with "git series format", or send a "please pull" request with
"git series req".  git-series knows the base of your series, so you
don't need to count patches or find a commit hash to run rebase or
format.

If you're interested in trying git-series, see
https://github.com/git-series/git-series for installation instructions
and a "getting started" guide.

I've also documented the internal storage format of git-series at
https://github.com/git-series/git-series/blob/master/INTERNALS.md ,
including the details for how git-series ensures git can always reach,
push, and pull a series.

I'd welcome any feedback, whether on the interface and workflow, the
internals and collaboration, ideas on presenting diffs of patch series,
or anything else.

- Josh Triplett

[toc] | [next] | [standalone]

#1452375

On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
[snip]
> 
> I'd welcome any feedback, whether on the interface and workflow, the
> internals and collaboration, ideas on presenting diffs of patch series,
> or anything else.
> 

This looks awesome!

I've been working on some similar stuff for a while also.[1][2]

I'm particularly interested in trying to establish a standard for
storing review data in git. I've got a prototype for doing that[3],
and an example tool that uses it[4]. The tool is still incomplete/buggy though.

There seem to be a number of us trying to solve this in our different ways,
it would be great to coordinate our efforts.

The prototype library I have is partly the result of some discussion and work
with the Gerrit folks, since they were thinking about this problem
before I even started writing git-candidate, and solved it with Notedb.[5]

Let me know if you'd like to work together on this,
I've been considering taking the perl-notedb prototype and writing
a C library for it with bindings for other languages (i.e. Rust).

[1]: http://www.mail-archive.com/git%40vger.kernel.org/msg79461.html
[2]: http://www.mail-archive.com/git%40vger.kernel.org/msg80972.html

[3]: https://bitbucket.org/richardipsum/perl-notedb
[4]: https://bitbucket.org/richardipsum/git-candidate

[5]: https://storage.googleapis.com/gerrit-talks/summit/2015/NoteDB.pdf

[toc] | [prev] | [next] | [standalone]

#1452383

On Fri, Jul 29, 2016 at 11:10:11AM +0100, Richard Ipsum wrote:
> On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> [snip]
> > 
> > I'd welcome any feedback, whether on the interface and workflow, the
> > internals and collaboration, ideas on presenting diffs of patch series,
> > or anything else.
> > 
> 
> This looks awesome!
> 
> I've been working on some similar stuff for a while also.[1][2]
> 
> I'm particularly interested in trying to establish a standard for
> storing review data in git. I've got a prototype for doing that[3],
> and an example tool that uses it[4]. The tool is still incomplete/buggy though.

Looks promising, though!

> There seem to be a number of us trying to solve this in our different ways,
> it would be great to coordinate our efforts.

These definitely seem like a family of related problems.  I'd like to
use git-series as a format for storing iterations on things like GitHub
pull-requests or Gerrit patch versions (in the latter case, overcoming
Gerrit's limitations on only handling one patch at a time).  Integrating
reviews with that seems helpful.

> The prototype library I have is partly the result of some discussion and work
> with the Gerrit folks, since they were thinking about this problem
> before I even started writing git-candidate, and solved it with Notedb.[5]
> 
> Let me know if you'd like to work together on this,

I'd love to.

I'll be presenting git-series at LinuxCon North America; will you be
there by any chance?  If not, perhaps we could meet by IRC or some other
medium and talk about this family of problems.

I hope to use git notes with git-series in the future, by putting
another gitlink under the git-series for notes related to the series.
I'd intended that for more persistent notes; putting them in the series
solves some of the problems related to notes refs, pushing/pulling, and
collaboration.  Using notes for review comments makes sense as well,
whether in a series or in a separate ref.

> I've been considering taking the perl-notedb prototype and writing
> a C library for it with bindings for other languages (i.e. Rust).

A C library based on libgit2 seems like a good idea; ideally the
bindings could interoperate with git2-rs.  (Alternatively, Rust can
*export* a C interface, so you could write directly with git2-rs. :) )

One of the items on my long-term TODO list is a completely federated
GitHub; I've been looking at other aspects of that, but federated
reviews/comments/etc seem critical to that as well.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1452395

On Fri, Jul 29, 2016 at 04:04:26AM -0700, Josh Triplett wrote:
[snip]
> 
> These definitely seem like a family of related problems.  I'd like to
> use git-series as a format for storing iterations on things like GitHub
> pull-requests or Gerrit patch versions (in the latter case, overcoming
> Gerrit's limitations on only handling one patch at a time).  Integrating
> reviews with that seems helpful.

Worth noting here that Gerrit's one patch per change format isn't
intrinsic to Notedb, since we just need to track the sha we want
to merge and optionally the branch we intend to merge into.

> 
> > The prototype library I have is partly the result of some discussion and work
> > with the Gerrit folks, since they were thinking about this problem
> > before I even started writing git-candidate, and solved it with Notedb.[5]
> > 
> > Let me know if you'd like to work together on this,
> 
> I'd love to.
> 
> I'll be presenting git-series at LinuxCon North America; will you be
> there by any chance?  If not, perhaps we could meet by IRC or some other
> medium and talk about this family of problems.

Cool :)

I didn't plan to be at LinuxCon North America,
but I can certainly send contact details out of band.

> 
> I hope to use git notes with git-series in the future, by putting
> another gitlink under the git-series for notes related to the series.
> I'd intended that for more persistent notes; putting them in the series
> solves some of the problems related to notes refs, pushing/pulling, and
> collaboration.  Using notes for review comments makes sense as well,
> whether in a series or in a separate ref.

Sounds interesting, can you explain how this works in more detail?
I ended up solving the push/pull issue with a custom merge driver
that effectively runs the Notedb parser on each side of the merge
and emits the union of the two sets of change notes.

> 
> > I've been considering taking the perl-notedb prototype and writing
> > a C library for it with bindings for other languages (i.e. Rust).
> 
> A C library based on libgit2 seems like a good idea; ideally the
> bindings could interoperate with git2-rs.  (Alternatively, Rust can
> *export* a C interface, so you could write directly with git2-rs. :) )

Certainly a fair alternative, though it may arguably be safer to write
the C and export to other languages, as cool as Rust looks it's not
established the way C is, so may be a slightly riskier foundation,
in my view.

And ofcourse in C we have native access to libgit2.

> 
> One of the items on my long-term TODO list is a completely federated
> GitHub; I've been looking at other aspects of that, but federated
> reviews/comments/etc seem critical to that as well.
> 

I agree.

[toc] | [prev] | [next] | [standalone]

#1452403

On Fri, Jul 29, 2016 at 01:44:44PM +0100, Richard Ipsum wrote:
> On Fri, Jul 29, 2016 at 04:04:26AM -0700, Josh Triplett wrote:
> > I hope to use git notes with git-series in the future, by putting
> > another gitlink under the git-series for notes related to the series.
> > I'd intended that for more persistent notes; putting them in the series
> > solves some of the problems related to notes refs, pushing/pulling, and
> > collaboration.  Using notes for review comments makes sense as well,
> > whether in a series or in a separate ref.
> 
> Sounds interesting, can you explain how this works in more detail?

The tree within a git-series commit includes a blob "cover" for the
cover letter, a gitlink "base" for the base commit, and a gitlink
"series" for the top of the series.  I could add a gitlink "notes",
which acts like a notes ref; then, each version of the series would have
its own notes ref.  As with the series, git-series would track the
"history of history"; since git-notes themselves use git history to
store a set of notes, git-series would store the history of the notes.
So if you add, remove, or change a note, git-series would track that as
a change to the notes ref.  If you merge/rebase/etc the notes ref to
merge notes, git-series would track that too.  A different series would
have a different set of notes, so you wouldn't be limited to
one notes ref per repository.

This doesn't solve the problem of merging notes, but it *does* mean you
have a full history of the changes to notes, not just the notes
themselves.

Something similar might work for the Gerrit notesdb.

> > > I've been considering taking the perl-notedb prototype and writing
> > > a C library for it with bindings for other languages (i.e. Rust).
> > 
> > A C library based on libgit2 seems like a good idea; ideally the
> > bindings could interoperate with git2-rs.  (Alternatively, Rust can
> > *export* a C interface, so you could write directly with git2-rs. :) )
> 
> Certainly a fair alternative, though it may arguably be safer to write
> the C and export to other languages, as cool as Rust looks it's not
> established the way C is, so may be a slightly riskier foundation,
> in my view.

I was mostly joking there.  Rust makes that potentially reasonable,
unlike most languages that can consume but not easily provide a C API,
but that doesn't make it the ideal solution quite yet. :)

> And ofcourse in C we have native access to libgit2.

Right.

> > One of the items on my long-term TODO list is a completely federated
> > GitHub; I've been looking at other aspects of that, but federated
> > reviews/comments/etc seem critical to that as well.
>
> I agree.

[toc] | [prev] | [next] | [standalone]

#1452837

On Fri, Jul 29, 2016 at 06:00:55AM -0700, Josh Triplett wrote:
> On Fri, Jul 29, 2016 at 01:44:44PM +0100, Richard Ipsum wrote:
> > On Fri, Jul 29, 2016 at 04:04:26AM -0700, Josh Triplett wrote:
> > > I hope to use git notes with git-series in the future, by putting
> > > another gitlink under the git-series for notes related to the series.
> > > I'd intended that for more persistent notes; putting them in the series
> > > solves some of the problems related to notes refs, pushing/pulling, and
> > > collaboration.  Using notes for review comments makes sense as well,
> > > whether in a series or in a separate ref.
> > 
> > Sounds interesting, can you explain how this works in more detail?
> 
> The tree within a git-series commit includes a blob "cover" for the
> cover letter, a gitlink "base" for the base commit, and a gitlink
> "series" for the top of the series.  I could add a gitlink "notes",
> which acts like a notes ref; then, each version of the series would have
> its own notes ref.  As with the series, git-series would track the
> "history of history"; since git-notes themselves use git history to
> store a set of notes, git-series would store the history of the notes.
> So if you add, remove, or change a note, git-series would track that as
> a change to the notes ref.  If you merge/rebase/etc the notes ref to
> merge notes, git-series would track that too.  A different series would
> have a different set of notes, so you wouldn't be limited to
> one notes ref per repository.
> 
> This doesn't solve the problem of merging notes, but it *does* mean you
> have a full history of the changes to notes, not just the notes
> themselves.
> 
> Something similar might work for the Gerrit notesdb.
> 

Okay I think there is a misunderstanding, Notedb is based on notes,
but they're not used in the same way as git-notes,
an example will help explain what I mean,

For a candidate 'update_readme' we store the change/candidate/whatever
metadata at refs/candidates/heads/up/update_readme/meta which is analogous
to Gerrit's notedb refs which uses something like refs/changes/34/1234/meta,
the prototype library I've written supports both forms and allows for some
flexibility in the naming of the prefix of the former type of ref
(so you may use refs/series/heads/up/update_readme/meta for example).

So the output of,
    git log -p refs/candidates/heads/up/update_readme/meta

gives

commit 38d0c182a46dc5a0f5d04ea0890e278b8e7a6eb6
Author: Richard Ipsum <richardipsum@fastmail.co.uk>
Date:   Sun Jul 24 16:59:16 2016 +0100

    Metadata update
    
    Patch-set: 1
    Status: merged

commit f45a396a156e121f923321e7530e74746e10bdb8
Author: Richard Ipsum <richardipsum@fastmail.co.uk>
Date:   Sun Jul 24 16:50:13 2016 +0100

    Vote on patch set 1
    
    
    
    Label: CodeReview=+1
    Patch-set: 1

commit b74eb15c1847d3bb28618c738c8ebc3412b6935a
Author: Richard Ipsum <richardipsum@fastmail.co.uk>
Date:   Sun Jul 24 16:48:11 2016 +0100

    Update our README to reflect reality
    BranchCommit; 59c46c9fa03725308779841f95ad71e7ccdb919c
    
    Branch: master
    Commit: 761d8da03a10b63b0b1e3cf97ffd7ececb09e3d6
    Patch-set: 1
    Status: new
    Subject: update_readme

This Notedb history is the result of the following git-candidate invocations

    git candidate create update_readme -m "Update our README to reflect reality"
    git candidate vote +1
    (use whatever git commands you like to merge the change)
    git candidate close update_readme

Basically any change made to a change in Notedb is recorded in a git history.

The format is explained in some more detail here[1].

[1]: https://storage.googleapis.com/gerrit-talks/summit/2015/NoteDB.pdf

[toc] | [prev] | [next] | [standalone]

#1452481

On Fri, Jul 29, 2016 at 5:44 AM, Richard Ipsum
<richard.ipsum@codethink.co.uk> wrote:
>>
>> These definitely seem like a family of related problems.  I'd like to
>> use git-series as a format for storing iterations on things like GitHub
>> pull-requests or Gerrit patch versions (in the latter case, overcoming
>> Gerrit's limitations on only handling one patch at a time).  Integrating
>> reviews with that seems helpful.
>
> Worth noting here that Gerrit's one patch per change format isn't
> intrinsic to Notedb, since we just need to track the sha we want
> to merge and optionally the branch we intend to merge into.

Note that Gerrit started to lose the "one patch at a time" notion.
It is possible to at least submit multiple changes coupled together
(even across project boundaries) via the topic. Some sort of cover
letter is missing though, that could be used e.g. for the merge commit.

[toc] | [prev] | [next] | [standalone]

#1452836

On Fri, Jul 29, 2016 at 09:59:08AM -0700, Stefan Beller wrote:
> On Fri, Jul 29, 2016 at 5:44 AM, Richard Ipsum
> <richard.ipsum@codethink.co.uk> wrote:
> >>
> >> These definitely seem like a family of related problems.  I'd like to
> >> use git-series as a format for storing iterations on things like GitHub
> >> pull-requests or Gerrit patch versions (in the latter case, overcoming
> >> Gerrit's limitations on only handling one patch at a time).  Integrating
> >> reviews with that seems helpful.
> >
> > Worth noting here that Gerrit's one patch per change format isn't
> > intrinsic to Notedb, since we just need to track the sha we want
> > to merge and optionally the branch we intend to merge into.
> 
> Note that Gerrit started to lose the "one patch at a time" notion.
> It is possible to at least submit multiple changes coupled together
> (even across project boundaries) via the topic. Some sort of cover
> letter is missing though, that could be used e.g. for the merge commit.

Potentially my misuse of the format but git-candidate puts the cover
letter into the body of the commit message before the footers begin,
for each new patchset added to the change. This has the advantage
that you can track each version of the cover letter,
since there's one per patchset.

[toc] | [prev] | [next] | [standalone]

#1452987

On Fri, Jul 29, 2016 at 12:10 PM, Richard Ipsum
<richard.ipsum@codethink.co.uk> wrote:
> On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> [snip]
>>
>> I'd welcome any feedback, whether on the interface and workflow, the
>> internals and collaboration, ideas on presenting diffs of patch series,
>> or anything else.
>
> This looks awesome!
>
> I've been working on some similar stuff for a while also.[1][2]
>
> I'm particularly interested in trying to establish a standard for
> storing review data in git. I've got a prototype for doing that[3],
> and an example tool that uses it[4]. The tool is still incomplete/buggy though.

There is also git-appraise (https://github.com/google/git-appraise)
written in Go to store code review data in Git.
It looks like it stores its data in git notes and can be integrated
with Rust (https://github.com/Nemo157/git-appraise-rs).

> There seem to be a number of us trying to solve this in our different ways,
> it would be great to coordinate our efforts.

Yeah, I agree.

[toc] | [prev] | [next] | [standalone]

#1453032

Christian Couder <christian.couder@gmail.com> wrote:
> On Fri, Jul 29, 2016 at 12:10 PM, Richard Ipsum
> <richard.ipsum@codethink.co.uk> wrote:
> > On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> > [snip]
> >>
> >> I'd welcome any feedback, whether on the interface and workflow, the
> >> internals and collaboration, ideas on presenting diffs of patch series,
> >> or anything else.

> > I'm particularly interested in trying to establish a standard for
> > storing review data in git. I've got a prototype for doing that[3],
> > and an example tool that uses it[4]. The tool is still incomplete/buggy though.
> 
> There is also git-appraise (https://github.com/google/git-appraise)
> written in Go to store code review data in Git.
> It looks like it stores its data in git notes and can be integrated
> with Rust (https://github.com/Nemo157/git-appraise-rs).

I'm not convinced another format/standard is needed besides the
email workflow we already use for git and kernel development.

Rather, better ways to archive/search the emails is desirable.
Fortunately, commit titles are rather unique :)

I started archiving the git ML with public-inbox (which uses git):

  https://public-inbox.org/git/20160710004813.GA20210@dcvr.yhbt.net/T/

It can be easy to search by Subject (commit titles):

  https://public-inbox.org/git/?q=s:%22more+archives+of+this+list%22

Search (currently Xapian) will be tuned to parse things like
filenames and diffs to allow searching within those.  It is
already somewhat email-aware, such as deprioritizing quoted
text; and having a code repository browser with mail archive
integration is in the works.

I also see the reliance on an after-the-fact search engine
(which can be tuned/replaced) as philosophically inline with
what git does, too, such as not having rename tracking and
doing delayed deltafication.

Email also has the advantage of having existing tooling, and
being (at least for now) federated without a single point of
failure.

vger.kernel.org can still be a major point of failure, which is
why the "archives first" approach of public-inbox favors readers
pulling messages over NNTP/HTTP/git (and maybe soon, POP3).

[toc] | [prev] | [next] | [standalone]

#1453044

On Mon, Aug 01, 2016 at 07:55:54AM +0000, Eric Wong wrote:
> Christian Couder <christian.couder@gmail.com> wrote:
> > On Fri, Jul 29, 2016 at 12:10 PM, Richard Ipsum
> > <richard.ipsum@codethink.co.uk> wrote:
> > > On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> > > [snip]
> > >>
> > >> I'd welcome any feedback, whether on the interface and workflow, the
> > >> internals and collaboration, ideas on presenting diffs of patch series,
> > >> or anything else.
> 
> > > I'm particularly interested in trying to establish a standard for
> > > storing review data in git. I've got a prototype for doing that[3],
> > > and an example tool that uses it[4]. The tool is still incomplete/buggy though.
> > 
> > There is also git-appraise (https://github.com/google/git-appraise)
> > written in Go to store code review data in Git.
> > It looks like it stores its data in git notes and can be integrated
> > with Rust (https://github.com/Nemo157/git-appraise-rs).
> 
> I'm not convinced another format/standard is needed besides the
> email workflow we already use for git and kernel development.

Not all projects use a patches-by-email workflow, or want to.  To the
extent that tools and projects use some other workflow, standardizing
the format they use to store patch reviews (including per-line
annotations, approvals, test results, etc) seems preferable to having
each tool use its own custom format.

> I also see the reliance on an after-the-fact search engine
> (which can be tuned/replaced) as philosophically inline with
> what git does, too, such as not having rename tracking and
> doing delayed deltafication.

Storing review data in git doesn't mean it needs to end up in the
history of the project itself; it can use after-the-fact annotations on
a commit.

> Email also has the advantage of having existing tooling, and
> being (at least for now) federated without a single point of
> failure.

Storing review data in git makes it easy to push and pull it, which can
provide the basis for a federated system.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1453084

On Mon, Aug 01, 2016 at 01:59:29AM -0700, Josh Triplett wrote:
> On Mon, Aug 01, 2016 at 07:55:54AM +0000, Eric Wong wrote:
[snip]
> > 
> > I'm not convinced another format/standard is needed besides the
> > email workflow we already use for git and kernel development.
> 
> Not all projects use a patches-by-email workflow, or want to.  To the
> extent that tools and projects use some other workflow, standardizing
> the format they use to store patch reviews (including per-line
> annotations, approvals, test results, etc) seems preferable to having
> each tool use its own custom format.

I concur, for better or for worse many projects have abandoned
mailing lists in favour of github, gerrit, gitlab and the like.
The problem being, with the exception of gerrit, most of these
tools store review data in sql databases, which is bad for obvious reasons.

> 
> > I also see the reliance on an after-the-fact search engine
> > (which can be tuned/replaced) as philosophically inline with
> > what git does, too, such as not having rename tracking and
> > doing delayed deltafication.
> 
> Storing review data in git doesn't mean it needs to end up in the
> history of the project itself; it can use after-the-fact annotations on
> a commit.

Exactly.

[toc] | [prev] | [next] | [standalone]

#1453480

Josh Triplett <josh@joshtriplett.org> wrote:
> On Mon, Aug 01, 2016 at 07:55:54AM +0000, Eric Wong wrote:
> > Christian Couder <christian.couder@gmail.com> wrote:
> > > On Fri, Jul 29, 2016 at 12:10 PM, Richard Ipsum
> > > <richard.ipsum@codethink.co.uk> wrote:
> > > > On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> > > > [snip]
> > > >>
> > > >> I'd welcome any feedback, whether on the interface and workflow, the
> > > >> internals and collaboration, ideas on presenting diffs of patch series,
> > > >> or anything else.
> > 
> > > > I'm particularly interested in trying to establish a standard for
> > > > storing review data in git. I've got a prototype for doing that[3],
> > > > and an example tool that uses it[4]. The tool is still incomplete/buggy though.

> > I'm not convinced another format/standard is needed besides the
> > email workflow we already use for git and kernel development.
> 
> Not all projects use a patches-by-email workflow, or want to.  To the
> extent that tools and projects use some other workflow, standardizing
> the format they use to store patch reviews (including per-line
> annotations, approvals, test results, etc) seems preferable to having
> each tool use its own custom format.

I think standardizing on email conventions (such as what we
already do with format-patch, request-pull, S-o-b trailers) would
be a step in this direction and a good step to take.

But yeah, I also hope git adopters can somehow be convinced to
also adopt the workflow that built git itself.

> > I also see the reliance on an after-the-fact search engine
> > (which can be tuned/replaced) as philosophically inline with
> > what git does, too, such as not having rename tracking and
> > doing delayed deltafication.
> 
> Storing review data in git doesn't mean it needs to end up in the
> history of the project itself; it can use after-the-fact annotations on
> a commit.

Right.  So on public-inbox.org/git today, one could search for
after-the-fact annotations based on commit titles and maybe
exact commit ID matches.

A future goal might be to get search indexing working on commit
ID substrings.  So finding references to commit
deadbeefcafe01234567890123467890abcdef00 could be done by
searching for "commit deadbeefcafe" or even a shorter ID, and
the following results could still be returned:

  1. commit deadbeefcafe broke my cat feeder
  2. commit deadbeef killed my cow

> > Email also has the advantage of having existing tooling, and
> > being (at least for now) federated without a single point of
> > failure.
> 
> Storing review data in git makes it easy to push and pull it, which can
> provide the basis for a federated system.

Every public-inbox exposed over HTTP(S) is git clonable[1], so
it's possible to push/pull or have developers merge/combine
inboxes with index-only operations.  There's no UI for that,
yet, and having a working tree checked out is inefficient with
300K uncompressed mails...

But there needs to be way to message others about the existence
of new pushes/pull-requests/reviews/etc; including users
unable to clone or host 800M git repos; so that messaging
system might as well be email.



[1] git clone --mirror https://public-inbox.org/git/
    That's not efficient, yet, though, at around 800M when the
    gzipped fast-export dump is around half that:
    https://public-inbox.org/git/20160710034745.GA20270@dcvr.yhbt.net/T/#u

[toc] | [prev] | [next] | [standalone]

#1453275

On 07/29/2016 12:40 AM, Josh Triplett wrote:
> I'd like to announce a project I've been working on for a while:
>
> git-series provides a tool for managing patch series with git, tracking
> the "history of history". git series tracks changes to the patch series
> over time, including rebases and other non-fast-forwarding changes. git
> series also tracks a cover letter for the patch series, formats the
> series for email, and prepares pull requests.

Just as an FYI, I wouldn't be surprised if there's some overlap, or 
potential for merging of tools, between this tool and the "patman" tool 
that's part of the U-Boot source tree:

http://git.denx.de/?p=u-boot.git;a=blob;f=tools/patman/README;h=e36857dedea1d0dbafa41732aaf9bf0988d63f38;hb=HEAD

[toc] | [prev] | [next] | [standalone]

#1453423

On Mon, Aug 01, 2016 at 09:14:54AM -0600, Stephen Warren wrote:
> On 07/29/2016 12:40 AM, Josh Triplett wrote:
> > I'd like to announce a project I've been working on for a while:
> > 
> > git-series provides a tool for managing patch series with git, tracking
> > the "history of history". git series tracks changes to the patch series
> > over time, including rebases and other non-fast-forwarding changes. git
> > series also tracks a cover letter for the patch series, formats the
> > series for email, and prepares pull requests.
> 
> Just as an FYI, I wouldn't be surprised if there's some overlap, or
> potential for merging of tools, between this tool and the "patman" tool
> that's part of the U-Boot source tree:
> 
> http://git.denx.de/?p=u-boot.git;a=blob;f=tools/patman/README;h=e36857dedea1d0dbafa41732aaf9bf0988d63f38;hb=HEAD

Interesting tool; thanks for the link.

As far as I can tell from that documentation, patman doesn't track old
versions of a patch series; you rebase to modify patches or change
patman tags (embedded in commit messages), and nothing preserves the
previous version.  And it tracks the cover letter and similar in one of
the commit messages in the series, so previous versions of that don't
get saved either.  If you wanted to track the history of your changes,
you'd have to use branch names or similar.

In addition, tracking metadata in commit messages only works with a
patches-by-mail workflow where the messages get processed when
generating patches; that doesn't work for please-pull workflows.

patman does have quite a few interesting ideas, though.  git-series
needs some way of handling To/Cc addresses for patches and the cover
letter (beyond just scripts/get_maintainer.pl), and more automatic
handling of series versioning (v2, v3, ...) and associated series
changelogs.  Suggestions welcome.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1455982

On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> 
> I'd welcome any feedback, whether on the interface and workflow, the
> internals and collaboration, ideas on presenting diffs of patch series,
> or anything else.
> 

One other nice thing I've noticed about this tool is the
way series behave like regular git branches: I specify the name
of the series and from then on all other commands act on that
series until told otherwise.

git-appraise looks as though it might also have this behaviour.
I think it's a nice way to do it, since you don't generally
perform more than one review simultaneously. So I may well
use this idea in git-candidate if it's okay. :)

I haven't found time to use the tool to do any serious review
yet, but I'll try and post some more feedback when I do.

Hope this helps,
Richard

[toc] | [prev] | [next] | [standalone]

#1456810

On Wed, Aug 03, 2016 at 08:12:02PM +0100, Richard Ipsum wrote:
> On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> > I'd welcome any feedback, whether on the interface and workflow, the
> > internals and collaboration, ideas on presenting diffs of patch series,
> > or anything else.
> 
> One other nice thing I've noticed about this tool is the
> way series behave like regular git branches: I specify the name
> of the series and from then on all other commands act on that
> series until told otherwise.

Thanks; I spent a while thinking about that part of the workflow.  I
save the current series as a symbolic ref SHEAD, and everything operates
on SHEAD.  (I should probably add support for running things like "git
series log" or "git series format" on a different series, because right
now "until told otherwise" doesn't include a way to tell it otherwise.)

One fun detail that took a couple of iterations to get right: I keep
separate "staged" and "working" versions per-series, so even with
outstanding changes to the cover letter, base, or series, you can always
detach or checkout another series without losing anything.  If you
switch back, all your staged and unstaged changes will remain staged and
unstaged where you left them.  That solves the "checkout a different
series with modifications to the current series" case.

> git-appraise looks as though it might also have this behaviour.
> I think it's a nice way to do it, since you don't generally
> perform more than one review simultaneously. So I may well
> use this idea in git-candidate if it's okay. :)

By all means.  For a review tool like git-candidate, it seems like you'd
want even more contextual information, to make it easier to specify
things like "comment on file F line L".  For instance, what if you
spawned the diff to review in an editor, with plenty of extra context
and a file extension that'll cause most editors to recognize it as a
patch (and specifically a git-candidate patch to allow specialized
editor modes), and told people to add their comments after the line they
applied to?  When the editor exits successfully, you can scan the file,
detect the added lines, and save those as comments.  You could figure
out the appropriate line by looking for the diff hunk headers and
counting line numbers.

If you use a format-patch diff that includes the headers and commit
message, you could also support commenting on those in the same way.
Does the notedb format support commenting on those?

> I haven't found time to use the tool to do any serious review
> yet, but I'll try and post some more feedback when I do.

Thanks!

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1459479

On Thu, Aug 04, 2016 at 12:40:58PM -1000, Josh Triplett wrote:
> On Wed, Aug 03, 2016 at 08:12:02PM +0100, Richard Ipsum wrote:
> > On Thu, Jul 28, 2016 at 11:40:55PM -0700, Josh Triplett wrote:
> > > I'd welcome any feedback, whether on the interface and workflow, the
> > > internals and collaboration, ideas on presenting diffs of patch series,
> > > or anything else.
> > 
> > One other nice thing I've noticed about this tool is the
> > way series behave like regular git branches: I specify the name
> > of the series and from then on all other commands act on that
> > series until told otherwise.
> 
> Thanks; I spent a while thinking about that part of the workflow.  I
> save the current series as a symbolic ref SHEAD, and everything operates
> on SHEAD.  (I should probably add support for running things like "git
> series log" or "git series format" on a different series, because right
> now "until told otherwise" doesn't include a way to tell it otherwise.)

Apologies for this delayed response,
I needed time to gather my thoughts,
and also to fix the perl libgit2 binding to allow me to use
your symbolic ref suggestion. :p

Though it turns out that libgit2 doesn't currently allow
me to write arbitrary data to a symbolic ref as git-symbolic-ref(1) will,
so this still needs to be fixed somehow.

> 
> One fun detail that took a couple of iterations to get right: I keep
> separate "staged" and "working" versions per-series, so even with
> outstanding changes to the cover letter, base, or series, you can always
> detach or checkout another series without losing anything.  If you
> switch back, all your staged and unstaged changes will remain staged and
> unstaged where you left them.  That solves the "checkout a different
> series with modifications to the current series" case.

Cool

> 
> > git-appraise looks as though it might also have this behaviour.
> > I think it's a nice way to do it, since you don't generally
> > perform more than one review simultaneously. So I may well
> > use this idea in git-candidate if it's okay. :)
> 
> By all means.  For a review tool like git-candidate, it seems like you'd
> want even more contextual information, to make it easier to specify
> things like "comment on file F line L".  For instance, what if you
> spawned the diff to review in an editor, with plenty of extra context
> and a file extension that'll cause most editors to recognize it as a
> patch (and specifically a git-candidate patch to allow specialized
> editor modes), and told people to add their comments after the line they
> applied to?  When the editor exits successfully, you can scan the file,
> detect the added lines, and save those as comments.  You could figure
> out the appropriate line by looking for the diff hunk headers and
> counting line numbers.

I really like this idea, the current interface for commenting is a little
tedious I find.

> 
> If you use a format-patch diff that includes the headers and commit
> message, you could also support commenting on those in the same way.
> Does the notedb format support commenting on those?

Comments in notedb are just a git note keyed on the sha of the
commit being commented on, I'm not certain what advantage a format-patch
diff provides in this case?

I've been closely following the 'patch submission process' thread,
and given the discussion there I'm having doubts over the value
of comments in git-candidate vs the mailing list. It seems to me that
git-candidate has many of the disadvantages of Github/Gitlab when it
comes to comments, for example, there is no threading.

Also the system would be less open than the mailing list, since,
as it stands currently you would require push access to the repository
to comment on anything.

It may be worth reflecting that one reason some organisations
have switched away from mailing list reviews to Github/Gitlab is that
they provide patch tracking, where the mailing list provides none,
so patches there can be 'lost'. So instead of trying to reimplement
an entire Gerrit/Github/Gitlab ui on the commandline, I wonder whether
it would be sufficient to add the minimum functionality necessary
to provide git with native patch tracking, and leave comments for the
mailing list. Ofcourse this is exactly what git-series seems to do,
so in some sense I may be advocating dropping my own work in favour of
improving git-series.

On the other hand, relying on the mailing list means that some of the
history of a series is left outside of the repository which is
anathema to the goal of git based/stored review, not least because
mail archives are centralised.
(which can obviously be problematic (as we've seen recently with gmane))

Maybe there's a better solution to this problem than git-candidate then,
maybe we can just invent some wonderful new subcommand that fetches
a mailing list archive into a git repo, for those that want that,
I don't know.

Out of interest, did you have any thoughts on Notedb itself with respect
to its suitability for git-series?

> 
> > I haven't found time to use the tool to do any serious review
> > yet, but I'll try and post some more feedback when I do.
> 
> Thanks!
> 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web