Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #268229 > unrolled thread

Committing git working tree with other git repos

Back to article view | Back to linux.debian.user

Contents

  Committing git working tree with other git repos Paul M Foster <paulf@quillandmouse.com> - 2024-03-13 16:10 +0100
    Re: Committing git working tree with other git repos john doe <johndoe65534@mail.com> - 2024-03-13 19:40 +0100
      Re: Committing git working tree with other git repos Paul M Foster <paulf@quillandmouse.com> - 2024-03-13 20:30 +0100
    Re: Committing git working tree with other git repos Tim Woodall <debianuser@woodall.me.uk> - 2024-03-13 22:50 +0100
      Re: Committing git working tree with other git repos Tim Woodall <debianuser@woodall.me.uk> - 2024-03-13 23:00 +0100
        Re: Committing git working tree with other git repos Tim Woodall <debianuser@woodall.me.uk> - 2024-03-14 20:50 +0100

#268229 — Committing git working tree with other git repos

Folks:

I have a /home/paulf/stow directory with contains subdirectories for each
of the packages whose dotfiles I want to manage, like:

/home/paulf/stow/alacritty

In each subdirectory, I have all the config files for that packages, under
git management. This means that the directory will look like this:

/home/paulf/stow/alacritty/.git
/home/paulf/stow/alacritty/.config/alacritty/alacritty.yml

This works well with stow (configs are now symlinks in $HOME).

I'd like to copy all of this to a git repo on gitlab. You would think you
could go to the ~/stow directory, "git init", then "git add" each
directory, and all is good. However, git looks inside the directories and
sees there are already .git directories there, and refuses to add the
directories and their contents to its repo. Instead, it wants you to use
"submodules", to wit:

git submodule add ./alacritty

This adds an *empty* alacritty subdirectory to the git repo, which isn't
useful.

I need a way to bring all these subdirectories and their contents under a
git repo so I can send it to gitlab. Any suggestions?

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

[toc] | [next] | [standalone]

#268239

On 3/13/24 16:04, Paul M Foster wrote:
> Folks:
>
> I have a /home/paulf/stow directory with contains subdirectories for each
> of the packages whose dotfiles I want to manage, like:
>
> /home/paulf/stow/alacritty
>
> In each subdirectory, I have all the config files for that packages, under
> git management. This means that the directory will look like this:
>
> /home/paulf/stow/alacritty/.git
> /home/paulf/stow/alacritty/.config/alacritty/alacritty.yml
>
> This works well with stow (configs are now symlinks in $HOME).
>
> I'd like to copy all of this to a git repo on gitlab. You would think you
> could go to the ~/stow directory, "git init", then "git add" each
> directory, and all is good. However, git looks inside the directories and
> sees there are already .git directories there, and refuses to add the
> directories and their contents to its repo. Instead, it wants you to use
> "submodules", to wit:
>
> git submodule add ./alacritty
>
> This adds an *empty* alacritty subdirectory to the git repo, which isn't
> useful.
>
> I need a way to bring all these subdirectories and their contents under a
> git repo so I can send it to gitlab. Any suggestions?
>
>

Sometime, learning something new is better than trying to get your own
way! ;^)

I can only suggest you to dig into Git submodules.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#268242

On Wed, Mar 13, 2024 at 07:29:39PM +0100, john doe wrote:

> On 3/13/24 16:04, Paul M Foster wrote:
> > Folks:
> > 
> > I have a /home/paulf/stow directory with contains subdirectories for each
> > of the packages whose dotfiles I want to manage, like:
> > 
> > /home/paulf/stow/alacritty
> > 
> > In each subdirectory, I have all the config files for that packages, under
> > git management. This means that the directory will look like this:
> > 
> > /home/paulf/stow/alacritty/.git
> > /home/paulf/stow/alacritty/.config/alacritty/alacritty.yml
> > 
> > This works well with stow (configs are now symlinks in $HOME).
> > 
> > I'd like to copy all of this to a git repo on gitlab. You would think you
> > could go to the ~/stow directory, "git init", then "git add" each
> > directory, and all is good. However, git looks inside the directories and
> > sees there are already .git directories there, and refuses to add the
> > directories and their contents to its repo. Instead, it wants you to use
> > "submodules", to wit:
> > 
> > git submodule add ./alacritty
> > 
> > This adds an *empty* alacritty subdirectory to the git repo, which isn't
> > useful.
> > 
> > I need a way to bring all these subdirectories and their contents under a
> > git repo so I can send it to gitlab. Any suggestions?
> > 
> > 
> 
> Sometime, learning something new is better than trying to get your own
> way! ;^)
> 
> I can only suggest you to dig into Git submodules.

I've read through https://git-scm.com/book/en/v2/Git-Tools-Submodules , and
it appears that submodules aren't really made for what I'm trying to do.
According to the docs, a submodule is more or less a link to an external
code repo. When used properly with the proper commands, the remote code can
be reconstituted. However, if you read the above, this won't work for my
purposes. That's why I asked the original question.

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

[toc] | [prev] | [next] | [standalone]

#268246

On Wed, 13 Mar 2024, Paul M Foster wrote:

> Folks:
>
> I have a /home/paulf/stow directory with contains subdirectories for each
> of the packages whose dotfiles I want to manage, like:
>
> /home/paulf/stow/alacritty
>
> In each subdirectory, I have all the config files for that packages, under
> git management. This means that the directory will look like this:
>
> /home/paulf/stow/alacritty/.git
> /home/paulf/stow/alacritty/.config/alacritty/alacritty.yml
>
> This works well with stow (configs are now symlinks in $HOME).
>
> I'd like to copy all of this to a git repo on gitlab. You would think you
> could go to the ~/stow directory, "git init", then "git add" each
> directory, and all is good. However, git looks inside the directories and
> sees there are already .git directories there, and refuses to add the
> directories and their contents to its repo. Instead, it wants you to use
> "submodules", to wit:
>
> git submodule add ./alacritty
>
> This adds an *empty* alacritty subdirectory to the git repo, which isn't
> useful.
>
> I need a way to bring all these subdirectories and their contents under a
> git repo so I can send it to gitlab. Any suggestions?
>
> Paul
>
>

I don't know exactly how to do it but git filter-branch might get you
started.

You can use that to rewrite the history of each branch so they're now
deeper subdirectories.

Now add a remote with an empty commit on master. git rebase each repo
onto that and push as a branch.

Finally git rebase each branch onto master in turn.

If I've understood your need correctly this should not give any
conflicts.


If you don't want to mess with rewriting history then just regular git
mv, rebase onto common master and rebase each tree in turn. But this
will (IMO) make history harder to understand and any future rebase
cleanups will likely be a disaster.

Tim.

[toc] | [prev] | [next] | [standalone]

#268247

On Wed, 13 Mar 2024, Tim Woodall wrote:

> On Wed, 13 Mar 2024, Paul M Foster wrote:
>
>> Folks:
>> 
>> I have a /home/paulf/stow directory with contains subdirectories for each
>> of the packages whose dotfiles I want to manage, like:
>> 
>> /home/paulf/stow/alacritty
>> 
>> In each subdirectory, I have all the config files for that packages, under
>> git management. This means that the directory will look like this:
>> 
>> /home/paulf/stow/alacritty/.git
>> /home/paulf/stow/alacritty/.config/alacritty/alacritty.yml
>> 
>> This works well with stow (configs are now symlinks in $HOME).
>> 
>> I'd like to copy all of this to a git repo on gitlab. You would think you
>> could go to the ~/stow directory, "git init", then "git add" each
>> directory, and all is good. However, git looks inside the directories and
>> sees there are already .git directories there, and refuses to add the
>> directories and their contents to its repo. Instead, it wants you to use
>> "submodules", to wit:
>> 
>> git submodule add ./alacritty
>> 
>> This adds an *empty* alacritty subdirectory to the git repo, which isn't
>> useful.
>> 
>> I need a way to bring all these subdirectories and their contents under a
>> git repo so I can send it to gitlab. Any suggestions?
>> 
>> Paul
>> 
>> 
>
> I don't know exactly how to do it but git filter-branch might get you
> started.
>
> You can use that to rewrite the history of each branch so they're now
> deeper subdirectories.
>
> Now add a remote with an empty commit on master. git rebase each repo
> onto that and push as a branch.
>
> Finally git rebase each branch onto master in turn.
>
> If I've understood your need correctly this should not give any
> conflicts.
>
>
> If you don't want to mess with rewriting history then just regular git
> mv, rebase onto common master and rebase each tree in turn. But this
> will (IMO) make history harder to understand and any future rebase
> cleanups will likely be a disaster.
>
> Tim.
>
>

Even easier if you don't need git checkout to give everything in a
single tree:

Just push each project as a separate branch to a common remote then git
worktree add to flatten it back to your current structure

git fetch --all will be your friend here to see which branches on a
remote have changed.

[toc] | [prev] | [next] | [standalone]

#268272

  On Wed, 13 Mar 2024, Paul M Foster wrote:

> Folks:
> 
> I have a /home/paulf/stow directory with contains subdirectories for each
> of the packages whose dotfiles I want to manage, like:
> 
> /home/paulf/stow/alacritty
> 
> In each subdirectory, I have all the config files for that packages, under
> git management. This means that the directory will look like this:
> 
> /home/paulf/stow/alacritty/.git
> /home/paulf/stow/alacritty/.config/alacritty/alacritty.yml
> 
> This works well with stow (configs are now symlinks in $HOME).
> 
> I'd like to copy all of this to a git repo on gitlab. You would think you
> could go to the ~/stow directory, "git init", then "git add" each
> directory, and all is good. However, git looks inside the directories and
> sees there are already .git directories there, and refuses to add the
> directories and their contents to its repo. Instead, it wants you to use
> "submodules", to wit:
> 
> git submodule add ./alacritty
> 
> This adds an *empty* alacritty subdirectory to the git repo, which isn't
> useful.
> 
> I need a way to bring all these subdirectories and their contents under a
> git repo so I can send it to gitlab. Any suggestions?
> 
> Paul
> 
>

So I thought this was a rather interesting exercise and tried it on one
of my repos that contains etckeeper files, one branch per machine.

I came up with this script (beware if your branches have weird
characters in the names or something, there's limited quoting/escaping
here.)

# clone the repo (I'm assuming you've managed to merge all your repos
# into one with a separate branch for each. I started from this so I've
# not got commands to do it but it shouldn't be hard, just add a
# commonremote and push to a named branch for each existing repo)
git clone -n git@einstein:/configs.git
cd configs

# Create a new branch with a completely empty commit at the root
# This must not match any existing branch.
rbp=rebasepoint
tree=$( git hash-object -wt tree --stdin < /dev/null )
commit=$( git commit-tree -m 'root commit' $tree )
git branch $rbp $commit
git checkout $rbp

# Don't know how to stop this one getting created but we need to delete
# it to simplify the rest. I expected git clone -n to not create this!
git branch -d master

# First map all the commits in each branch on the remote into a
# subdirectory of the branch name on my (relatively low power) machine
# this maps about 30 objects per second.
# This has a very long line with subtle quoting - take care when
# cutting/pasting.
for i in $( git branch -r | grep -v HEAD ); do
   echo $i
   git filter-branch -f --index-filter 'git ls-files -s | sed "s:\t\"*:&'"$i"'/:" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && if [ -e "$GIT_INDEX_FILE.new" ]; then mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"; fi' -- $i
done

# Now rebase each branch onto the previous one (Note that we're starting
# with rebasepoint that we created above)
# This gets progressively slower on my machine, not exactly sure why.
for i in $( git branch -r | grep -v HEAD ); do
   git branch --track ${i#origin/} $i
   git rebase $rbp ${i#origin/}
   rbp=${i#origin/}
done

# The branch you are on at this point should be a branch that combines
# all of the upstream branches


# If anything goes wrong, just delete the configs directory and start
# again. You are changing nothing on the upstream unless/until you
# decide to push.


tim@dirac:~/git/flatten/configs (xen3)$ ls origin/
aptmirror17  citrix17       dirac       ipmi17  ntp17        wiki17
aptmirror19  cups17         einstein    ipmi19  ntp19        wiki19
asterisk17   debootstrap17  firewall17  ipmi2   proxy17      xen17
asterisk19   debootstrap19  firewall19  mail17  proxy19      xen19
backup17     debootstrap2   firewall2   mail19  rpi          xen2
bind17       dhcp17         imap17      master  rpi-flat17a  xen3
bind19       dhcp19         imap19      mtd19   victoria17

HTH.

Tim.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web