Enhanced version of Portage client driver

[casso]

Hi, I've created an enhanced client side driver for portage/ebuild clients. It is a major re-write of the old Portage driver. The driver supports the following enhancements:

• Allow version auto or any
• Allow building packages from source with binpkgonly = False option
• Allow for calculating dependencies of specified packages with get_deps = True option
• Reasonable verbose mode logging output
• Fallback to old driver's behaviour by default

Introduction

For a Gentoo Linux machine, dependency calculation must be done client side, and it must be done after these steps:

• portage tree is synchronised
• any overlays are synchronised
• make.conf file is up to date
• any files under /etc/portage are up to date (specifically portage.use and portage.keywords)

For the same dependency checks to be implemented server side and work successfully, the architecture would most likely need to be the same on the host server. Additionally the host server would need to have the same state as the client regarding the above mentioned points. It is certainly much easier and makes much more sense to check this client side.

To check dependencies client side, the client lies about the list of extra packages. The FindExtraPackages function has been written to check what dependencies would be required to install all specified packages from the literal configuration. If any packages remain, these will be listed as extra packages.

Auto/any version support

This has been implemented with an additional function called _CheckForUpdates. This function will check all packages specified in the literal configuration. If a package has version auto it will be checked if an update can be performed. If a package has version any and is not installed it will be checked if an update can be performed.

Update checking is done in a single pass. To do this the full list of packages is fed to emerge (specifically emerge --update --newuse --nodeps) and the output is checked to see if the corresponding package exists. If it does exist, the version it will be installed as is noted.

Dependency checking via FindExtraPackages

The function FindExtraPackages was not implemented in the previous driver. To support dependency checking, this function lies about the list of extra packages.

The function takes the list of specified packages, checks if each package is installed then checks the dependencies via emerge --emptytree of the installed packages. If any packages that are currently installed are not in the returned list, the package is marked as an extra package.

Since running this function is costly time wise, the function will only perform the calculation twice; once at start-up and once after package installation/removal. The more default mode of operation would call this function once for every modified bundle after package installation/removal.

If dependency checking is not to be performed, the parent FindExtraPackages function will be used instead. The parent function will be called at startup and once for every modified bundle.

New Install function

The original driver did not implement the install driver. The new install driver checks if the specified version is explicitly specified. If it is not, the driver substitutes the version that should be installed (checked originally via _CheckForUpdates) and calls the parent Install function.

Modifications to VerifyPackage

The package verification method has been modified to account for versions of type auto and any. In all cases the package checks (done via equery check) will be performed on the installed version if requested. For packages of version auto, the installed version will be checked, but the package may fail verification because an update is required.

Configuration file options

The following two options have been added to the configuration file under the Portage section

• binpkgonly
  • Type: Boolean
  • Default: True
  • Description: Will use emerge --binpkgonly to install packages. When false, use options specified in EMERGE_DEFAULT_OPTS
• get_deps
  • Type: Boolean
  • Default: False
  • Description: Selects if the list of extra packages will not include packages required to install those packages specified in the configuration

Suggested methods of use

This information should be added to the documentation.

The new driver when included into the bcfg2 client would ideally be run from a cron job. The cron job would be as follows:

emerge --sync
layman --sync-all  # If required
emerge --update --deep world
bcfg2 -Qb critical
bcfg2 -r all
revdep-rebuild
emerge --depclean
eclean-dist -D

In the above example, the bundle critical would contain the bcfg2.conf file, the make.conf file, portage.use and portage.keywords. Running this bundle first will make sure that these changes are applied before any checking of packages is done.

If using binpkgonly=False, it is highly recommended to adjust EMERGE_DEFAULT_OPTS to something like --usepkgonly and have pre-built packages across your site. For build hosts (machines that would be compiling these packages for distribution), options like --newuse --update --deep could be used.

It is also recommended to look into appropriate PORTAGE_ELOG values in make.conf when compiling packages.

[casso]

Made a couple of minor changes:

• The Install function was attempting to return a value from the parent Install function. This is no longer the case.
• Made sure the system package is always specified in the list of packages for getting dependencies.
• Removed some fallback checking in the FindExtraPackages? function since it is no longer used

[solj]

Thanks! Going to give this a shot on my end before committing the changes.

[solj]

This looks good. The auto/any functionality should actually be done on the server side (via the Packages plugin) in order to keep consistency with the rest of the plugins/client tools.

The client tools in Bcfg2 do no processing of their own configuration (in general). So, the configuration sent to the client from the server should be complete. Let me know if you have any questions. I'll work on putting together a modified version of your other updates.

[casso]

I've made a few modifications to the driver since I've posted it last. The Install function previously installed a fixed package version in all cases. It now installs only a fixed version if a fixed version is specified, otherwise it will simply install the required package name and let the package manager determine the version.

Please re-read the introduction I originally posted regarding why I chose to have the client determine dependencies and package versions, as opposed to the server side. For the server to calculate the dependency tree and correct versions, it would need:

• To most likely match in architecture (e.g.: x86 vs x86-64)
• To have the same /etc/make.conf configuration
• To have the same /etc/portage/* files (primarily package.use and package.keywords)
• To have the same /etc/make.profile symlink
• To have the same overlays specified
• To have overlays and main portage tree all completely in sync between client and server

I'm not saying that it couldn't be done, but it certainly adds quite a substantial lot of effort. These files would need to be determined for their correct details at runtime on the server before binding particular package versions.

[solj]

I've added a trimmed down version of this in 8470a15940309428d9286f2ef6372749dd14bddf.

Replying to casso:

I've made a few modifications to the driver since I've posted it last. The Install function previously installed a fixed package version in all cases. It now installs only a fixed version if a fixed version is specified, otherwise it will simply install the required package name and let the package manager determine the version.

Please re-read the introduction I originally posted regarding why I chose to have the client determine dependencies and package versions, as opposed to the server side. For the server to calculate the dependency tree and correct versions, it would need:

• To most likely match in architecture (e.g.: x86 vs x86-64)
• To have the same /etc/make.conf configuration
• To have the same /etc/portage/* files (primarily package.use and package.keywords)
• To have the same /etc/make.profile symlink
• To have the same overlays specified
• To have overlays and main portage tree all completely in sync between client and server

While I read your introduction initially, it still doesn't completely correspond with the way Bcfg2 works. There are problems which can (and often do) arise when you are unable to determine client state. Having the client configuration depend on the state of the client is (potentially) dangerous.

That being said, your tool looks like it could work fine. Also, notice that you can rename your tool to, say, Portageng.py and simply drop it in the Client/Tools? directory of your installed version to use it.

I'm not saying that it couldn't be done, but it certainly adds quite a substantial lot of effort. These files would need to be determined for their correct details at runtime on the server before binding particular package versions.

It's definitely not 'easy' to implement, but it's nicer to send the client a _complete_ specification so that you can more easily debug any configuration problems. The same argument was had over rpm/deb dependency resolution prior to Packages.