All available architectures in the current builder are collected and packages for architectures which are not supported
in the current builder instance are culled.
This is a much easier way to eliminate architectures that depend only on a given debian suite than
specifying the dists: key in the package file.
The dists keys are still used for packages which have additional restrictions (like kernels too old to compile with
newer toolchains, even if the architecture itself is supported).
When a package is added to the REPO any duplicates are supposed to be removed automatically.
The (package-name, version, arch) was determined by splitting the (fullpath) name by underscore.
If an underscore was present in any of the parent directories then the split did not yield
the correct information. As a result any duplicate packages were not removed and were instead
left in the REPO.
This fix splits the package basename instead of the package path.
You may now clone your ONL worktree as JEFF_IS_STUPID and it should work now.
- Fix the OnlPackageRepo lock.
The lock for the package repo has always been broken for some requests due to the nested nature of some of the calls.
This would cause the repo lock to be release prematurely in the middle of operations.
This becomes obvious once large parallel package builds are enabled.
The OnlPackageRepo class has been renamed to OnlPackageRepoLocked() and all locking has been removed.
All operations performed by the OnlPackageRepoLocked() class are now assumed to be non-concurrent.
The OnlPackageRepo class is now a wrapper around the OnlPackageRepoLocked class with acquires the lock
prior to requesting any operations. This change is transparent to the callers but ensures
that no nested locks (and premature unlocks) are performed.
- Improve the parallel makefile generation.
- Set pipefail to that targets fail even with the log tee.
- Allow packages with broken dependencies to set their stage explicitly.
- Parallel package builds require that the package cache be treated as read-only.
- Packages must specify their package prerequisites properly in their PKG file.
- In order to detect and correct this situation packages will no longer
be implicitly built as part of calls to onlpm find and link operations.
We may be able to relax this restriction once the build is tight.
- Package Cache Corruption
- If the package cache is corrupt (usually due to an aborted operation) then it we will
automatically rebuild it instead of just whining about it.
Specifying useful package fields has been historically pretty lazy since most systems have been built as self-contained images without dynamic package updates.
This commit hopes to improve the situation.
The recommended package organization for ONL and for systems built with ONL as the base follow the pattern of $project/packages/base/$arch/* and $project/packages/platforms/$vendor/$arch/
The heirarchy of directories in which the packages reside also provide a natural inheritence heirarchy for debian package fields.
This property is now exploited to allow default package keys to be specified base on their location in the filesystem heirarchy as follows:
Before a PKG.yml file is evaluated its parent directory heirarchy is traversed looking for either or both of these files:
PKG_DEFAULTS.yml -- This is an onlyaml file which can specify the values of package keys.
PKG_DEFAULTS -- If this file is executable it is expected to output valid yaml containing package keys.
All of these values are then combined in reverse order to provide a default dict for the package.
Example usages:
- Fields global to all packages in the project (like Vendor, Maintainer, Version [for coherent builds], maintainer, etc) can be specified in the top-most PKG_DEFAULTS.
- Archicture can be specified in the $arch/PKG_DEFAULTS.
- Platform Vendors can specify their information at the top of their platform directory.
By specifying your own PKG_DEFAULTS you can coexist the ONL versions and your own version without ineheriting any of the defaults from ONL itself.
These are only the defaults for keys not specified in the PKG file. You can still obviously override every field in each package declaration.
The ONL package files will be updated in a separate commit to use this new hierarchical approach.
Package version numbers, copyrights, and general information fields have all
Packages can now restrict the debian distribution for which they can build using the new 'dists' package key.
This is a csv list of distribution codenames. If the currently building distribution is not in the list then
the package is ignored.
The 'links' section specifies symbolic links which should be added to the package in key : value format.
The key is the source of the link. It must be exist in the filesystem already (as part of the 'files' section)
and be relative to the root of the filesystem.
The value is the name of the link, and can be relative or absolute to the final filesystem.
For example, given that a package produces the real binary "/usr/bin/foobar" and you want /usr/bin/foobar-link -> /usr/bin/foobar
it will be specified as follows:
links:
/usr/bin/foobar : /usr/bin/foobar-link
Some complicated service dependencies will fail to configure
under normal circumstances because invoke-rc.d will kill service
start based on policy-rc.d but this is not taken into account when
configuring services that are dependent on that service.
The dependency check fails prior to the policy-rc.d check and as a result
some packages will remain unconfigured after we have finished constructing
the root filesystem. This is ok for most cases but precludes dynamic modification
of that filesystem post-creation and will even cause
those unconfigured services to be restarted the first time a new package is
installed.
Checking policy-rc.d first in the postinst script allows the configuration
step to complete for all services and their dependents to fix these problems.
The package caches contain paths relative to the distro name (from the $BUILD_DIR variable).
As such the cache can become invalid when switching in-place between distro containers.
Keeping the cache instanced per-container makes sure they don't contaminate each other.
