Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #62282 > unrolled thread

Testing git-debrebase/dgit in the linux git repo

Back to article view | Back to linux.debian.kernel

Contents

  Testing git-debrebase/dgit in the linux git repo Ben Hutchings <ben@decadent.org.uk> - 2018-10-04 00:10 +0200
    Re: Testing git-debrebase/dgit in the linux git repo Ben Hutchings <ben@decadent.org.uk> - 2018-10-04 01:20 +0200
      Re: Testing git-debrebase/dgit in the linux git repo Ian Jackson <ijackson@chiark.greenend.org.uk> - 2018-10-04 03:30 +0200
        Re: Testing git-debrebase/dgit in the linux git repo Ian Jackson <ijackson@chiark.greenend.org.uk> - 2018-10-04 03:40 +0200
        Re: Testing git-debrebase/dgit in the linux git repo Bastian Blank <waldi@debian.org> - 2018-10-14 20:50 +0200
          Re: Testing git-debrebase/dgit in the linux git repo Ian Jackson <ijackson@chiark.greenend.org.uk> - 2018-10-14 23:30 +0200
          Re: Testing git-debrebase/dgit in the linux git repo Ian Jackson <ijackson@chiark.greenend.org.uk> - 2018-10-15 00:30 +0200
            Re: Testing git-debrebase/dgit in the linux git repo Bastian Blank <waldi@debian.org> - 2018-10-16 21:50 +0200
      Re: Testing git-debrebase/dgit in the linux git repo Sean Whitton <spwhitton@spwhitton.name> - 2018-10-04 17:10 +0200
      Re: Testing git-debrebase/dgit in the linux git repo Bastian Blank <waldi@debian.org> - 2018-10-17 13:00 +0200

#62282 — Testing git-debrebase/dgit in the linux git repo

[Multipart message — attachments visible in raw view] — view raw

As discussed in yesterday's meeting, I've been investigating git-
debrebase and dgit as tools for maintaining the linux source package
with upstream included.  It seems like they could make it a lot easier
to add and rebase patches, while maintaining some patch history and
fast-forward branches (unlike a regular git rebase).

Both tools are in the dgit package, and there is a tutorial in the
dgit-maint-debrebase(7) manual page.

Using these tools requires three main changes to the git repository:

1. The upstream code, with full history, is merged into Debian
branches.

This makes the git repository substantially larger.  This may be a
problem for some users checking out development versions.  We might
need to modify instructions in debian-kernel-handbook and elsewhere to
recommend a clone with limited depth.

Currently we merge between the sid and master branches, but the
corresponding upstream branches are not (and should not be) merged.  We
may need to implement a custom merge strategy to make this work.  It
sems that git-debrebase does not (yet) deal well with merges; see the
description of the "--experimntal-merge-resolution" option in its
manual page.

2. Debian changes to upstream code are applied as commits.  The
debian/patches directory is created and committed by dgit only as
necessary.

This means that "git cherry-pick", "git am" etc. should work.  We will
still need to add DEP-3 pseudo-headers to the commit message though. 
We can use a "Gbp-Pq: Topic ..." trailer in the commit message to
organise changes into topics, which are mapped to subdirectories.

I wrote a check-commits script (paralleling check-patches.sh) which
checks for the expected DEP-3 pseudo-headers in Debian commits to the
upstream code.  check-patches.sh will only operate on featureset
patches, which still have to be managed as patches-in-git.

I also updated the genpatch-* scripts and makefiles to cope with this.

3. All generated files in the source package will be included in Debian
branches.

dgit requires that the git tree corresponding to some version of a
Debian source package exactly matches the unpacked package, with the
exception of the .pc directory for a 3.0 (quilt) package.

The only generated files which really need to be present in the source
package are debian/control and debian/tests/control.  We also need to
include debian/control.md5sum so that a stale debian/control file can
be detected.  I made changes to debian/bin/gencontrol.py and
debian/rules to enable this behaviour.  I also changed the way
debian/changelog is checksummed so that debian/control.md5sum won't
change so often.

---

A dgit-suitable version of the master branch now exists, named dgit-
master.  This is experimental, so DO NOT USE THIS BRANCH to build and
upload source packages.  But please do try making changes to this
branch in parallel to your changes to the master branch.

All the changes I described above are only on this branch, not the
master branch.

The initial conversion from debian/patches to git commits is done using
gbp-pq, which can lose headers (bug #905934).  I used a locally patched
version for the conversion.

Ben.

-- 
Ben Hutchings
Sturgeon's Law: Ninety percent of everything is crap.

[toc] | [next] | [standalone]

#62283

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2018-10-03 at 23:08 +0100, Ben Hutchings wrote:
[...]
> Currently we merge between the sid and master branches, but the
> corresponding upstream branches are not (and should not be) merged.  We
> may need to implement a custom merge strategy to make this work.  It
> sems that git-debrebase does not (yet) deal well with merges; see the
> description of the "--experimntal-merge-resolution" option in its
> manual page.
[...]

git-debrebase(5) goes into more detail about this.  Basically it
doesn't work after any merge commit where the new tree is not identical
to one of the parents.  So a merge from master to sid would be OK (the
result should match master even if it's not fast-forward) but a merge
in the other direction would not.  But worse than that, a non-fast-
forward merge from a remote tracking branch to the corresponding local
branch would also break it.

So I can't see how git-debrebase is usable for team-maintained packages
yet.  A "git pull" that merges is not safe, and a "git pull" that
rebases is also not safe because you must use "git debrebase" for all
rebases.

I thought I was told that team maintenance should be OK.  Did I
misunderstand?  Am I missing something?

Ben.

-- 
Ben Hutchings
Sturgeon's Law: Ninety percent of everything is crap.

[toc] | [prev] | [next] | [standalone]

#62284

Ben Hutchings writes ("Re: Testing git-debrebase/dgit in the linux git repo"):
> git-debrebase(5) goes into more detail about this.  Basically it
> doesn't work after any merge commit where the new tree is not identical
> to one of the parents.  So a merge from master to sid would be OK (the
> result should match master even if it's not fast-forward) but a merge
> in the other direction would not.  But worse than that, a non-fast-
> forward merge from a remote tracking branch to the corresponding local
> branch would also break it.
> 
> So I can't see how git-debrebase is usable for team-maintained packages
> yet.  A "git pull" that merges is not safe, and a "git pull" that
> rebases is also not safe because you must use "git debrebase" for all
> rebases.
> 
> I thought I was told that team maintenance should be OK.  Did I
> misunderstand?  Am I missing something?

How big a problem this is depends how much your team works in
parallel.

I do have some experimental support for merge resolution.  It ought to
deal reasonably well with cases where what has been merged is the
packaging, rather than the delta queues.  Ie, if only one side has
edited the patch queue.  It'll probably also cope well with addition
of independent new upstream-files commits to the end of the delta
series.

If it can't cope because both sides edited the delta queues in a way
it can't cope with, leaves a pile of refs containing pieces of the
wreckage, which an expert could fix up.

If you want to try this out on some actual existing merges in your
existing history, you can do it by passing
  --experimental-merge-resolution
on the command line.  If this is successful it will generate a merge
commit in your breakwater which new enough git-debrebase should be
willing to recognise and not try to rewrite, even without that option.

There are other options for resolving this kind of thing, including
using `git pull --rebase' more often, but they are also not
brilliant.

Anticipating this question is why I wrote the merge support but I have
not yet found anyone who needed it.  So as a result I don't know how
well it works in practice.

Nor do I know what the best way to handle failed delta queue merges
is.  Internally right now the algorithm tries to: identify a common
upstream commit; use gbp pq to convert each delta queue as patches;
use git merge to merge the patches textually; use a sort of graph
theory like algorithm to merge the series files; and apply the result
to the identified upstream.  If this all works then it's pretty
seamless.

If it doesn't work the most obvious reason is patch queue merge
failure, or application failure, which you can fix up by checking out
the wreckage refs, wrestling with conflicts in diff files or whatever,
and eventually telling gdr about an appropriate resolution.

I think in the general case merging delta queues looks something like
this example [1]

      Ul--1-2'-3"-4-5       Ul
     /                     /  \
   Ub---1-2-3        +   Ub    Um    =>   Um--2??-5-9-3"'-4-8
     \                     \  /
      Ur--5-9-3'-8          Ur

and this is (a) horrid (b) an open research problem.

I don't even know of a really good way to present a problem like this
to the user, so the tool can say "you guys did a crazy thing pls fix".

I hope this is illuminating.  If you want to try this merge feature I
think you'd be the first user and I would be happy to ... give you
fairly intensive support :-).

Regards,
Ian.

[1] Ul, Ur, Ub are the left right and base upstreams; Um is some
merged upsream which someone presumably supplies.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

[toc] | [prev] | [next] | [standalone]

#62285

Ian Jackson writes ("Re: Testing git-debrebase/dgit in the linux git repo"):
> If you want to try this out on some actual existing merges in your
> existing history, you can do it by passing
>   --experimental-merge-resolution
> on the command line.  If this is successful it will generate a merge
> commit in your breakwater which new enough git-debrebase should be
> willing to recognise and not try to rewrite, even without that option.

Hrm, I see the manpage warning is a bit overblown.  In particular

  branch structures that require the use of this same option by other
  people.

is not true.  (Or at least not suppose to be.)  I think mangling of
your source code is unlikely, too.  The merge code does have a couple
of test cases that prove it works in at least one case, and that it is
in principle possible to resolve a conflict.

I think I wrote the manpage before I wrote the tests, when the feature
had basically not been run.


If I knew something about what kinds of things are likely to have
occurred on the two branches that you're merging, I could be more
helpful about whether the merge algorithm would work for you.

I could also think about how to deal with your merge cases.  I suspect
that dealing with merges automatically is possible in a number of
(fairly common) specific kinds of scenario, even though writing a
single algorithm to solve the whole problem in all of those cases is
hard.

Ian.

[toc] | [prev] | [next] | [standalone]

#62366

Hi Ian

On Thu, Oct 04, 2018 at 02:25:17AM +0100, Ian Jackson wrote:
> How big a problem this is depends how much your team works in
> parallel.

We are using merge requests in gitlab, to review our work.  This means
we have merges all over the place.

I tried using git-debrebase on one of my other packages.  To mimic
merge request I used merges to introduce new upstream versions.  It did
not survive a git debrebase new-upstream, as the resulting tree is not
longer a fast forward to master and it actually removed commits.

See https://salsa.debian.org/waldi/lvm2-gitdebrebase-test/network/master

To get to this state I used the following commands:

% git checkout -b feature/2.02.177 master
% git debrebase new-upstream 2.02.177
% git checkout master
% git merge --no-ff feature/2.02.177
% git checkout -b feature/2.02.179 master
% git debrebase new-upstream 2.02.179

I tried the whole thing a second time, had to "git reset --hard" a few
times, but now git debrebase does not even manage to use
"convert-from-gbp", the patches vanish.  Not sure what this is about.

I'm really missing from the documentation how a git tree using debrebase
should look like, where the merges should be.

Bastian

-- 
She won' go Warp 7, Cap'n!  The batteries are dead!

[toc] | [prev] | [next] | [standalone]

#62369

Bastian Blank writes ("Re: Testing git-debrebase/dgit in the linux git repo"):
> We are using merge requests in gitlab, to review our work.  This means
> we have merges all over the place.

I think it will be necessary for me to write some kind of script to
convert your existing history into something I can use as test cases
for gdr.

> I tried using git-debrebase on one of my other packages.  To mimic
> merge request I used merges to introduce new upstream versions.

That would not work at all.  Sorry if I gave the impression that it
would be a sensible thing to try.

> I'm really missing from the documentation how a git tree using debrebase
> should look like, where the merges should be.

Well, without the experimental merge feature, there are supposed to be
none made by the user.  See `OTHER MERGES' in git-debrebase(5).  The
experimental merge feature is undocumented I'm afraid, so that's why
you can't find it in the docs.

*With* it enabled, the merges that are expected are between two gdr
interchange branches.  Ie, two people both do legal git-debrebase
things, and they just git merge at some point.  Both parents must be
gdr branches.

If one side wants to incorporate a new upstream, then that would be
   git-debrebase new-upstream
on that branch.  Then maybe merge the result with a gdr branch that
didn't have that done, and see if
  git-debrebase --experimntal-merge-resolution status
does anything useful.

gdr new-upstream on both sides will not work unless one of the
upstreams is ff of the other...

Regards,
Ian.

[toc] | [prev] | [next] | [standalone]

#62370

Bastian Blank writes ("Re: Testing git-debrebase/dgit in the linux git repo"):
> See https://salsa.debian.org/waldi/lvm2-gitdebrebase-test/network/master
> 
> To get to this state I used the following commands:
> 
> % git checkout -b feature/2.02.177 master
> % git debrebase new-upstream 2.02.177
> % git checkout master
> % git merge --no-ff feature/2.02.177
> % git checkout -b feature/2.02.179 master
> % git debrebase new-upstream 2.02.179

Wait, I looked at this again and I think I have misunderstood what you
tried.

Something like what you describe above might well work.  I cloned the
tree you mention and its master (80203aaf59da3b0c) looks in a
reasonable shape.  My git-debrebase is entirely happy with it.  But,
the merge you made looks like a psuedomerge because it is entirely
identical to onen of its parents.  So it's not a merge test case.
Also your `New upstream version 2.02.177' is some kind of tarball
import.  Is it not maintained in git upstream ?

I'm also confused about whether you are saying that the master branch
in that repo is the result, or the starting point, for the commands
you quote above.  If it is the result then I am confused because I see
no commit corresponding to new-upstream 2.02.179.

I can't repro what you did because I don't seem to be able to get the
right bits.  In particular, the tags like 2.02.177 are only available
from upstream or something ?  Your Vcs-Git header does not work for
me:
  zealot:lvm2-gitdebrebase-test> git-fetch -v vcs-git
  Username for 'https://gitlab.com':
?

Feel free to grab me on irc if that seems like it will work better...

Ian.

[toc] | [prev] | [next] | [standalone]

#62380

Hi

On Sun, Oct 14, 2018 at 11:21:36PM +0100, Ian Jackson wrote:
> Bastian Blank writes ("Re: Testing git-debrebase/dgit in the linux git repo"):
> > See https://salsa.debian.org/waldi/lvm2-gitdebrebase-test/network/master
> > 
> > To get to this state I used the following commands:
> > 
> > % git checkout -b feature/2.02.177 master
> > % git debrebase new-upstream 2.02.177
> > % git checkout master
> > % git merge --no-ff feature/2.02.177
> > % git checkout -b feature/2.02.179 master
> > % git debrebase new-upstream 2.02.179
> 
> Wait, I looked at this again and I think I have misunderstood what you
> tried.
> 
> Something like what you describe above might well work.  I cloned the
> tree you mention and its master (80203aaf59da3b0c) looks in a
> reasonable shape.  My git-debrebase is entirely happy with it.  But,
> the merge you made looks like a psuedomerge because it is entirely
> identical to onen of its parents.  So it's not a merge test case.
> Also your `New upstream version 2.02.177' is some kind of tarball
> import.  Is it not maintained in git upstream ?

It is a merge test case. Of cause one of the parents is identical to the
result of the merge, as the history of those merge requests will look
like:

  2-3    4-5
 /   \  /   \
1-----3*-----5*

> I'm also confused about whether you are saying that the master branch
> in that repo is the result, or the starting point, for the commands
> you quote above.  If it is the result then I am confused because I see
> no commit corresponding to new-upstream 2.02.179.

Take a look on the graph where the (not requested) "update changelog"
commits are.

> I can't repro what you did because I don't seem to be able to get the
> right bits.  In particular, the tags like 2.02.177 are only available
> from upstream or something ?  Your Vcs-Git header does not work for
> me:
>   zealot:lvm2-gitdebrebase-test> git-fetch -v vcs-git
>   Username for 'https://gitlab.com':

I have no idea what you are trying to do.  I gave you the URL to a test
project hosted on salsa.debian.org, clone it.

Bastian

-- 
Intuition, however illogical, is recognized as a command prerogative.
		-- Kirk, "Obsession", stardate 3620.7

[toc] | [prev] | [next] | [standalone]

#62291

[Multipart message — attachments visible in raw view] — view raw

Hello Ben,

On Thu 04 Oct 2018 at 12:15AM +0100, Ben Hutchings wrote:

> So I can't see how git-debrebase is usable for team-maintained
> packages yet.  A "git pull" that merges is not safe, and a "git pull"
> that rebases is also not safe because **you must use "git debrebase"
> for all rebases.**

Just to be clear about the recommendation to use git-debrebase for all
rebases: the reason for that recommendation is to avoid starting the
debrebase too early.

In cases where you haven't both edited the delta queue or performed a
debrebase, `git pull --rebase` won't start the rebase too early, so it's
fine to use it.

-- 
Sean Whitton

[toc] | [prev] | [next] | [standalone]

#62384

Hi Ben

On Thu, Oct 04, 2018 at 12:15:44AM +0100, Ben Hutchings wrote:
> I thought I was told that team maintenance should be OK.  Did I
> misunderstand?  Am I missing something?

I played with it a little bit in a smaller context.  What I understand
now is that git-debrebase uses the same branch name for trees in
different forms that by definition are not fast-forwardable.  The
documentation is a bit silent about it, or that certain operations just
transform trees in addition to the thing they are supposed to do.

Also I'm completely missing the step to create a final packageable
version of it.  It's burried deep in it and the documentation is again
silent.

It also happily removes commits that are published already and it thinks
are unnecessary, which is kind of a no-no.

Regards,
Bastian

-- 
Vulcans do not approve of violence.
		-- Spock, "Journey to Babel", stardate 3842.4

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web