Get Gentoo!
gentoo.org sites
gentoo.org Wiki Bugs Forums Packages
Planet Archives Sources
Infra Status
aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* catalyst: Clean up exception handlingpending/mattst88Matt Turner2021-01-276-100/+57
| | | | | | | Use 'raise from' where sensible, but a lot of the CatalystErrors are just trading one CatalystError for another, so remove them. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove fallback make.conf parsing codeMatt Turner2021-01-271-39/+3
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Use lazy % formatting in logging functionsMatt Turner2021-01-271-3/+3
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove unused importMatt Turner2021-01-271-1/+1
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Fix copy-and-paste mistakeMatt Turner2021-01-271-1/+1
| | | | | Fixes: 87b0588a ("catalyst: Add option to enter the chroot before building") Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Add option to enter the chroot before buildingMatt Turner2021-01-233-1/+22
| | | | | | | | With --enter-chroot, after the mounts and environment are set up, catalyst will drop you into a shell inside the chroot. Useful for hacking or debugging. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Deduplicate the common build_sequence stepsMatt Turner2021-01-236-20/+6
| | | | | | | This also provides a single location to add a function to all targets' build_sequence. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Deduplicate prepare_sequence assignmentsMatt Turner2021-01-236-37/+6
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Call config_profile_link for all targetsMatt Turner2021-01-232-4/+2
| | | | | | | | | | | | | | | | | | | | | | | This reverts the following two commits, which have no documented rationale. Revert "Set the profile by calling eselect." This reverts commit 90c03f9dc255ba89849e46490f9ead7ab3921950. Revert "Drop config_profile_link from the action_sequence for the generic stage." This reverts commit 3bd10159bf7cfe14b6d8a8218b94eca73be4c997. Doing so improves the code in two ways: 1) it makes prepare_sequence identical across all targets, which will allow deduplicating some code 2) it no longer calls eselect profile each time chroot-functions.sh is sourced (even for those targets that were still calling config_profile_link) Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Store references to functionsMatt Turner2021-01-237-124/+125
| | | | | | | ... rather than their names. This makes it possible for tooling to understand the code structure better. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Update the @changed-subslot set by defaultMatt Turner2021-01-231-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In portage commit 1789fdf2ee81 (Add @changed-subslot package set) I added this: the set of upgradable packages for which the highest visible version has a different subslot than the currently installed version. Updating the entire stage is expensive and unnecessary (since we're going to build the latest packages in stage1 and then rebuild everything in stage3). What we definitely do need to update in the original stage3 however, is any package that would trigger a subslot rebuild. For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc will link with libmpfr.so.4, but the latest version of dev-libs/mpfr will be built and libmpfr.so.6 included into the stage1. Since the old libmpfr.so.4 is not included in the stage1, gcc will not work, breaking subsequent stage builds. Our current options to update the seed are too large a hammer (e.g., "--update --deep --newuse @world" or "--update --deep --newuse --complete-graph --rebuild-if-new-ver gcc") and spend too much time updating seed stages for no gain beyond updating only packages for whom the subslot has changed. Bug: https://bugs.gentoo.org/739004 Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Update BINPKG_COMPRESS to new zstd defaultMatt Turner2021-01-231-0/+3
| | | | | | | | | | In portage commit 230595cf600c (Default BINPKG_COMPRESSION to zstd (bug 715108)) the default BINPKG_COMPRESSION setting was changed to zstd. But to retain compatibility with binpkg consumers, the default is unchanged for existing installations, so we need to update our BINPKG_COMPRESSION. Bug: https://bugs.gentoo.org/715108 Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Remove some obvious commentsMatt Turner2021-01-173-6/+0
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Remove unused 'enter' commandMatt Turner2021-01-175-18/+0
| | | | | | | Doesn't appear to have ever been wired up. I'm going to add something like what I think this was, but without the duplication. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Emerge stage1 with --implicit-system-deps=nMatt Turner2020-12-191-1/+1
| | | | | | | Since we're emerging into a separate ROOT, we can use --implicit-system-deps=n to gain some additional parallelism. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Remove make_destpath() functionMatt Turner2020-12-196-44/+23
| | | | | | | | It just read clst_root_path, exported it as ROOT, and ensured that a directory existed at $ROOT. Setting and resetting clst_* variables adds a lot of confusion, so just use ROOT directly. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Pass separate arguments to run_merge()Matt Turner2020-12-182-5/+5
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Remove EBEEP_IGNORE / EPAUSE_IGNOREMatt Turner2020-12-181-2/+0
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove some redundant set_stage_path()sMatt Turner2020-12-182-10/+0
| | | | | | | We only need to override root_path, and stage_path will be configured properly. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove set_dest_path()Matt Turner2020-12-182-16/+8
| | | | | | | root_path is always set, so half of this function was dead. The other half set the same thing as set_stage_path(). Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Set stage_path = chroot_path + root_pathMatt Turner2020-12-181-1/+2
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove dead commentMatt Turner2020-12-151-1/+0
| | | | | Fixes: 220df6a5d255 ("catalyst: Drop ancient portage support") Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Use interpreter with portage module for build.pyMatt Turner2020-12-151-2/+7
| | | | | Closes: https://bugs.gentoo.org/759685 Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Move ${buildpkgs} error checking closerMatt Turner2020-12-141-6/+6
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Make ${buildpkgs} an arrayMatt Turner2020-12-141-4/+3
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Rename clst_buildpkgs -> buildpkgsMatt Turner2020-12-141-3/+3
| | | | | | And remove the unnecessary export. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* doc: Make existing multi-overlay support more explicitFelix Bier2020-11-146-12/+18
| | | | | | | | The code already supports multiple repositories. This commit adapts the documentation to make this more explicit. Signed-off-by: Felix Bier <felix.bier@rohde-schwarz.com> Signed-off-by: Matt Turner <mattst88@gentoo.org>
* Move from PORTDIR_OVERLAY to repos.confFelix Bier2020-11-143-23/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes the following issues: * The PORTDIR_OVERLAY variable has been deprecated by Gentoo. With this commit, the variable is no longer written to the generated make.conf. Instead, a config file /etc/portage/repos.conf/<repo-name>.conf is generated for each overlay. The repo name is read from the overlay using the portage API. Internally, portage parses metadata/layout.conf and profiles/repo_name to obtain the name. References: https://wiki.gentoo.org/wiki//etc/portage/make.conf https://wiki.gentoo.org/wiki//etc/portage/repos.conf * All overlays were copied into the same target directory. If the same file name occurred in multiple overlays, the last overlay would overwrite all previous files with this name. In particular, only the metadata/layout.conf of the last overlay was retained, so it was not possible to reference the other overlays e.g. via the masters entry in the layout.conf or the portage-2 syntax for specifying a parent profile from another overlay. Also, this created problems when the overlays contained ebuilds for the same package, but with differing versions, because after copying, the target directory contained both versions of the ebuild but only the manifest file of the last overlay. With this commit, each overlay is copied into a separate sub-directory, e.g. /var/db/repos/<repo-name>. This directory is referenced via the location entry in the generated /etc/portage/repos.conf/<repo-name>.conf. Signed-off-by: Felix Bier <felix.bier@rohde-schwarz.com> Signed-off-by: Matt Turner <mattst88@gentoo.org>
* Ensure deep copying of config defaultsFelix Bier2020-11-142-2/+4
| | | | | | | | | | | | | | | | | | | | | | | This commit adds deep copying operations when initializing config objects from a default config. This prevents the config from being a shallow copy of the default, ensuring that modifications to the config do not modify the default. In particular, this fixes a check in write_make_conf, where the PORTDIR variable is supposed to be only written to the generated make.conf when a non-default repo_basedir is set in /etc/catalyst/catalyst.conf. This check is never satisfied, because confvalues is a shallow copy of confdefaults, therefore both will always hold the same value for repo_basedir. For self.mounts / MOUNT_DEFAULTS this problem can also be observed, the modifications done to self.mounts are also visible in MOUNT_DEFAULTS. I am not aware of any bugs due to this shallow copy, but I would prefer adding a deep copy to prevent future bugs, in case a comparision against the default mounts is ever needed. Signed-off-by: Felix Bier <felix.bier@rohde-schwarz.com> Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Restore root and cwd after exiting mount namespaceFelix Bier2020-11-141-0/+22
| | | | | | | | | | | | | | | | | | | | | This commit saves the file descriptor of /proc/self/{root,cwd} before entering into the new mount namespace. When restoring the previous mount namespace, it restores /proc/self/{root,cwd} based on the saved file descriptors. Without this change, catalyst cannot be run in a chroot when using the recent changes regarding mount namespaces: After the mount namespace has been exited, /proc/self/root points to the "/" of the host system, not the "/" of the chroot. Therefore, the cleanup phase of catalyst runs outside of the chroot. The code is similar to how nsenter(1) sets root and cwd: https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/sys-utils/nsenter.c#n452 Tested in a Gentoo chroot and in Gentoo VM (non-chroot). Signed-off-by: Felix Bier <felix.bier@rohde-schwarz.com> Signed-off-by: Matt Turner <mattst88@gentoo.org>
* targets: Expand extra_opts into separate wordsMatt Turner2020-11-021-1/+1
| | | | | | | I introduced this mistaken when reworking the patch referenced below. Fixes: 7e1ea3ac ("Build ISO with Joliet directory information") Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Drop unbind()Matt Turner2020-10-306-51/+1
| | | | | | | | | | | | | | mount_namespaces(7) says A mount ceases to be a member of a peer group when either the mount is explicitly unmounted, or when the mount is implicitly unmounted because a mount namespace is removed (because it has no more member processes). As a result, we can rely on exiting the mount namespace to unmount the bind mounts. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove mount_safety_check()Matt Turner2020-10-301-37/+0
| | | | | | | | | | | | | | | | mount_safety_check() exists to prevent bad things from happening if a previous catalyst invocation left bind mounts active in the chroot. E.g., a previous catalyst invocation is interrupted without unmounting the bind mounts. A new catalyst invocation runs and cleans the old chroot, which inadvertently deletes files outside of the chroot via the bind mounts. With all the mounts now inside a namespace, it is no longer possible to have mounts accessible outside the build sequence. In fact, I think this code has been unnecessary since commit e5a53e42 ("catalyst: create namespaces for building"). Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Remove kill_support_pids()Matt Turner2020-10-302-88/+4
| | | | | | | | | | | | | | | | mount_namespaces(7) says A mount ceases to be a member of a peer group when either the mount is explicitly unmounted, or when the mount is implicitly unmounted because a mount namespace is removed (because it has no more member processes). Now that the build sequence is executed in its own mount namespace, the mounts are implicitly unmounted when the last process in the namespace dies, meaning we don't need to try any funny business around cleaning up processes in order to unmount. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Run the build sequence in new mount namespaceMatt Turner2020-10-302-4/+5
| | | | | | | | | | | | | | Catalyst has a lot of code to unmount the bind mounts it's made, and then more to try harder when something fails. This is important because if bind mounts still exist within the chroot when clean up happens, files outside of the chroot on the host system can inadvertently be deleted. E.g., distfiles, binpkgs, kerncache. Running the build sequence (the steps that need bind mounts) within a mount namespace and exiting the mount namespace when finished ensures that clean up can never accidentally delete files outside the chroot. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Add and use namespace context managerMatt Turner2020-10-302-10/+39
| | | | | | | Wraps snakeoil's simple_unshare; returns to the previous namespaces on context exit. Will be used by the next commit. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Factor out run_sequence()Matt Turner2020-10-301-17/+23
| | | | | | | This is preparation for the next patch, which will run the build sequence in a separate mount namespace. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Split action_sequence into prepare/build/finishMatt Turner2020-10-307-22/+48
| | | | | | | | We want to run the "build" sequence in a different mount namespace from the "prepare" and "finish" sequences, so this splits action_sequence into those groupings. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Use .extend() and .append() for action_sequenceMatt Turner2020-10-306-15/+15
| | | | | | Ensures that we don't overwrite and lose some settings. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Move action_sequence out of self.settings[]Matt Turner2020-10-307-21/+22
| | | | | | | | This self.settings[] dictionary is very similar to the god object anti-pattern. Moving action_sequence out it starts us down the road of fixing this. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Use libmount for handling mountsMatt Turner2020-10-301-22/+35
| | | | | | | | | | Handle the mounts/unmounts in all in process rather than shelling out (pun intended!) to an external program. While we're here, change some log.notice to log.debug since those cases are normal and expected. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Rewrite ismount() to use libmountMatt Turner2020-10-303-10/+8
| | | | | | | | libmount is provided by util-linux, so this adds a dependency on sys-apps/util-linux[python]. A later patch will make more extensive use of this API. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Replace pathcompare()Matt Turner2020-10-301-17/+7
| | | | | | Modern Python allows us to do this in a much cleaner way. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* Makefile: Cleanup doc/subarches.generated.xmlW. Trevor King2020-10-291-2/+2
| | | | | | | | | | Since its addition in fa940f7 (Migrate man page to Asciidoc, generate list of subarches, add Makefile including "dist" target, 2011-06-24), doc/make_subarch_table_guidexml.py has also generated an XML version of the supported subarches. Add the XML file to DOC_SIDE_EFFECTS so it gets removed by `make clean`. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* Build ISO with Joliet directory informationDaniel Cordero2020-10-241-2/+3
| | | | | | | | | Previous syslinux based CDs had this information. Necessary for direct kernel booting with virt-install --location. Fixes: 0a27a7a39a "targets: Use GRUB for BIOS boot" Signed-off-by: Daniel Cordero <catalyst@0xdc.io> Signed-off-by: Matt Turner <mattst88@gentoo.org>
* Update requirements for amd64/x86 ISO creationDaniel Cordero2020-10-241-2/+6
| | | | | | | xorriso uses mkisofs to create the ISO. Signed-off-by: Daniel Cordero <catalyst@0xdc.io> Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Don't delete /usr/share/zoneinfo from stage1Matt Turner2020-10-221-1/+2
| | | | | | | | I have no clue why we would want to (or even bother) delete the timezone data out of stage1. It's been this way since the initial catalyst 2.0 import. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Explicitly keep /etc/resolv.conf in stage4Matt Turner2020-10-221-1/+5
| | | | | | | | | | It was confusing to understand that /etc/resolv.conf was *kept* in stage4 because its set_cleanables() contained everything except it. This will simplify things if we want to add more common items to the cleanables list. Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Drop useless stage3 set_cleanables()Matt Turner2020-10-221-3/+0
| | | | Signed-off-by: Matt Turner <mattst88@gentoo.org>
* catalyst: Drop stray vim modelineMatt Turner2020-10-221-2/+0
| | | | | | | This was the only one in catalyst. We should have an EditorConfig file instead. Signed-off-by: Matt Turner <mattst88@gentoo.org>