Commit Graph

138 Commits (fbd4214bb0727268aa535af5b600d38939ec8dbc)

Author SHA1 Message Date
Paul Spooren 941ec28b35 imagebuilder: Remove json_info_files/ before build
The folder `json_info_files` contains multiple JSON files which describe
created firmware images. The folder is not removed between builds as the
ImageBuilder does not use `image.mk`.

Not removing the JSON files result in a merged `profiles.json` file
containing entries for outdated or non-existing images.

This commit adds the `json_info_files/` cleanup step to the ImageBuilder
Makefile.

Signed-off-by: Paul Spooren <mail@aparcar.org>
4 years ago
Paul Spooren 4f38063640 imagebuilder: pass IB=1 on checking requirements
The patch 4a1a58a3  build, imagebuilder: Do not require libncurses-dev
was supposed to remove libncurses as a requirement for the ImageBuilder.
However as the IB=1 is only exported during building, not for checking
requirements, it did never actually work.

This commit export IB=1 to the requirement check.

Signed-off-by: Paul Spooren <mail@aparcar.org>
4 years ago
Paul Spooren 07449f692c build: refactor JSON info files to `profiles.json`
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").

They are useful for firmware wizards and script checking for
reproducibility.

Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.

This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.

Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.

Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.

As before, this creation is enabled by default only if `BUILDBOT` is set.

Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
4 years ago
Martin Schiller b204fcdb07 target/imagebuilder: use multi-thread support for xz compression
This speeds up the packing of the imagebuilder a lot:

 imagebuilder-T0.tar.xz real 0m25.199s user 2m45.967s sys 0m1.218s
 imagebuilder-T1.tar.xz real 2m02.543s user 2m02.418s sys 0m1.653s
 imagebuilder-T2.tar.xz real 1m03.684s user 1m59.931s sys 0m0.587s
 imagebuilder-T3.tar.xz real 0m48.033s user 2m02.904s sys 0m0.637s
 imagebuilder-T4.tar.xz real 0m38.963s user 2m15.521s sys 0m0.783s
 imagebuilder-T5.tar.xz real 0m37.994s user 2m21.461s sys 0m0.919s
 imagebuilder-T6.tar.xz real 0m39.524s user 2m48.115s sys 0m1.279s
 imagebuilder-T7.tar.xz real 0m34.061s user 2m45.097s sys 0m1.174s
 imagebuilder-T8.tar.xz real 0m27.286s user 2m55.449s sys 0m1.329s
 imagebuilder-T9.tar.xz real 0m25.205s user 2m44.894s sys 0m1.208s

To keep the output reproducible in any case, we enforce a minimum amount
of 2 threads.

Signed-off-by: Martin Schiller <ms@dev.tdt.de>
[refactored into reusable NPROC var, more verbose commit message]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 years ago
Paul Spooren 07926d7def imagebuilder: fix `make info` for empty SUPPORTED_DEVICES
For x86/64 (maybe more) target the SUPPORTED_DEVICES variable is empty
which causes the `&&` junction to fail, producing a non zero exit code.

Tested-by: Paul Spooren <mail@aparcar.org>
Fixed-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Paul Spooren <mail@aparcar.org>
5 years ago
Richard Musil 71ab2c9d17 imagebuilder: new DISABLED_SERVICES make variable
Adds a new variable DISABLED_SERVICES to ImageBuilder Makefile, which
defines a list of services (installed as /etc/init.d/*) to be disabled
during the build of a custom image (normally all are enabled).

It comes handy when a particular service should not be run under normal
circumstances, but should be ready in the image for situations when it
might be needed.

Signed-off-by: Richard Musil <risa2000x@gmail.com>
5 years ago
Daniel Golle d6fa04a437 IB: include SUPPORTED_DEVICES in 'make info' output
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
5 years ago
Paul Spooren 483498808e ib: show current revision based on $(REVISION)
This is useful in for the attendedsyupsgrade server (asu) to
distinguish between snapshot version. Currently asu can't tell devices
requesting a snapshot build if the same build is already installed.

Signed-off-by: Paul Spooren <mail@aparcar.org>
5 years ago
Paul Spooren f37afec866 ib: show unified target based on $(TARGETID)
Instead of showing a slightly more readable target like
"ar71xx (Generic)" print the more generic format "ar71xx/genric"

Signed-off-by: Paul Spooren <mail@aparcar.org>
5 years ago
Daniel Golle 13c379e5c6 ib: display whether profile comes with image metadata
Having image metadata (and signature) appended is a condition for
semi-automated sysupgrade, hence IB needs to be able to tell which
images will end up with metadata.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
5 years ago
Paul Spooren ad5c2897ec imagebuilder: manifest function show stderr
This really simplifies debugging, if a package is not found or a feed is
not reachable, a proper stderr is printed. Currently it would only say
`_call_manifest` failed.

Signed-off-by: Paul Spooren <mail@aparcar.org>
6 years ago
Paul Spooren 869b0d11db imagebuilder: add function to show manifest
Tested with 18.06.0-rc2/ar71xx/generic/tl-wdr4300-v1, image & list

This PR is based on the work of @fewckert[1] with slight improvements.

Add function `manifest` to show the manifest of the produced image,
before actually building it. The manifest contains an orderd list of
package name and version.

This is usefull to check package dependencies but also determine a
unique and reproducible image name before building the package. The
sysupgrade server[2] builds images on request with individual package
selection. To distignish between created images which contain differnt
packages, the EXTRA_IMAGE_NAME is set to a shortend hash of the
manifest's content. So far the image was renamed afterwards as the
manifests content was unknown, however this corrupts the signed
sha256sums. This patch allows a clean solution as to dtermine the
manifest in advance and set the EXTRA_IMAGE_NAME accordingly.

[1]: https://github.com/lede-project/source/pull/1591
[2]: https://github.com/aparcar/attendedsysupgrade-server

Signed-off-by: Paul Spooren <mail@aparcar.org>
6 years ago
Matthias Schiffer a02a69d5f8
imagebuilder: remove split patch dirs from imagebuilder archive
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
6 years ago
Matthias Schiffer 2fbf669730
imagebuilder: reuse rootfs preparation from rootfs.mk
In addition to removing redundant code, this fixes various issues in
IB-generated images that have been fixed in prepare_rootfs before,
including better handling of CONFIG_CLEAN_IPKG and enabling of initscripts
from FILES.

We also reuse the opkg macro and remove --force-... flags that have been
removed from rootfs.mk as well.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
6 years ago
Jo-Philipp Wich fd30187c87 imagebuilder: fix reference to removed VERSION_SED variable
Fixes: ff8e9a4ecb ("treewide: combine VERSION_SED and VERSION_SED_SCRIPT")
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
6 years ago
Zoltan HERPAI d2c06eb075 merge: etc: update remaining files
Signed-off-by: Zoltan HERPAI <wigyori@uid0.hu>
6 years ago
Zoltan HERPAI 7b5c989ab9 merge: targets: update image generation and targets
Signed-off-by: Zoltan HERPAI <wigyori@uid0.hu>
6 years ago
Daniel Golle d80d1b6c42 imagebuilder: don't rewrite package list output
No longer rewrite opkg list output in package_list function, remove
the awk call in the pipe (which was intended for a single specific
use-case).

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
7 years ago
Daniel Golle 1b555e1d2b imagebuilder: clean package_list
commit 19ac879954 (imagebuilder: add package_list function) introduced
a new function 'package_list' to the imagebuilder Makefile.
Unfortunately the package list was poluted by stdout noise of the
Makefile itself as well as opkg. Redirect those outputs to stderr to
make sure that the package_list returned doesn't contain progress
info output but really only packages.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
7 years ago
Paul Spooren 19ac879954 imagebuilder: add package_list function
The imagebuilder can now list all available packages by using make
package_list. This is usefull for scripts to retrieve a list of all
packages with versions (and size)

Signed-off-by: Paul Spooren <paul@spooren.de>
[daniel@makrotopia.org: fixed commit message]
7 years ago
Jo-Philipp Wich 37e7a1734f imagebuilder: fix bundling of DTS sources
Refer to LINUX_KARCH instead of ARCH when bundling DTS files in the image
builder tarball.

While we're at it, also dereference symbolic links when copying as some
kernel architectures contain symbolic links in their DTS directories.

This fixes aarch64 imagebuilders such as brcm2708/bcm2710 ones in particular
as the kernel refers to "aarch64" as "arm64" internally.

Ref: https://forum.lede-project.org/t/lede-image-builder-problem/3680

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years ago
Felix Fietkau 9467ce42da build: get rid of host.mk
Defined required host related variables in toplevel.mk instead

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Felix Fietkau 9dcb921d90 build: add buildbot specific config option for setting defaults
This can be used to tweak the buildbot behavior without having to change
buildbot's configuration.
It will also allow us to add more aggressive clean steps (e.g. on
toolchain changes), which would break developers' workflows if enable
by default.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Jo-Philipp Wich 0d1765b4ba imagebuilder: make submake invocations less verbose
Use silent make invocations for sub-makes like build_image or checksum to
avoid bloating the IB output with non-status info.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years ago
Jo-Philipp Wich 7d57db4d9b build: introduce STAGING_DIR_IMAGE
Introduce a new location STAGING_DIR_IMAGE which is intended to be used by
bootloader iamges and similar image-related artifacts.

This directory is guaranteed to be persistent across kernel upgrades which
might involve a removal of KERNEL_BUILD_DIR and is guranteed to be bundled
with the image builder.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years ago
Jo-Philipp Wich 6cb8e30837 imagebuilder: properly escape single quotes in device titles
The name "Plat'Home OpenBlocks AX3" causes the imagebuilders "make info"
command to fail with:

    bash: -c: line 0: syntax error near unexpected token `('
    bash: -c: line 0: `echo;  [...]'
    Makefile:99: recipe for target '_call_info' failed

Properly escape single quotes to avoid breaking the echo commands.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years ago
Felix Fietkau 619c8fa922 imagebuilder: remove existing debug kernel image
Reduces tarball size and improves build time

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Felix Fietkau d1514e8f84 imagebuilder: remove existing root filesystem images
Reduces tarball size and improves build time

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Jo-Philipp Wich 72d751cba9 build: rework library bundling
Rework the bundle-libraries.sh implementation to use a more robust approach
for executing host binaries through the shipped ELF loader and libraries.

The previous approach relied on symlinks pointing to a wrapper script which
caused various issues, especially with multicall binaries as the original
argv[0] name was not preserved through the ld.so invocation. Another down-
side was the fact that the actual binaries got moved into another directory
which caused executables to fail looking up resources with paths relative
to the executable location.

The new library wrapper implements the following improvements:

 - Instead of symlinks pointing to a common wrapper, each ELF executable
   is now replaced by a unqiue shell script which retains the original
   program name getting called

 - Instead of letting ld.so invoke the ELF executable directly, launch
   the final ELF binary through a helper program which fixes up the argv[0]
   argument for the target program

 - Support sharing a common location for the bundled libraries instead of
   having one copy in each directory containing wrapped binaries

Finally modify the SDK build to wrap the staging_dir and toolchain binaries
which allows to use the SDK on systems with a different glibc version.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years ago
Daniel Engberg 636a069c42 target/imagebuilder: Switch to xz compression instead of bz2
Switch to xz compression instead of using bz2.
Saves about 20% of total size (ar71xx)

Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
8 years ago
Daniel Dickinson 71370d2c55 target/{sdk,imagebuild}: Fix for symlink-tree
With symlink tree some directories are just symlinked which
means IB and SDK end up with a symlink instead of an actual
directory; this fixes the missing files by dereferencesing
the directories instead of copying the symlinks.

Signed-off-by: Daniel Dickinson <lede@cshore.thecshore.com>
8 years ago
Jo-Philipp Wich 27854a0a84 build: add checksum target
Add a new "checksum" make target which generates an sha256sums file over the
image files produced in bin/targets/ and automatically call it during make
world after the package index generation.

The advantage of this new target is that it is guaranteed to run after the
images, the SDK and the ImageBuilder archives have been generated to ensure
that they all end up in the checksum file. Fixes FS#51.

Uses sed to postprocess the OpenSSL digest output into an sha256sum command
compatible format.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years ago
Felix Fietkau 37e82e4e42 build: remove obsolete variables from opkg command
Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
Felix Fietkau 8265fdcc4d imagebuilder: strip DEVICE_ prefix from profiles (FS#55)
Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
Felix Fietkau 9ae952cf8c build: split scripts/metadata.pl into target-metadata.pl and package-metadata.pl
Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
Alexander Couzens 2e980479c1 IB/SDK/toolchain: use lower cases filenames
Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
8 years ago
Alexander Couzens 4a7c653400 IB/SDK/toolchain: use VERSION_DIST_SANITIZED instead of VERSION_DIST
VERSION_DIST can contains spaces which produces problems when used as file name

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
8 years ago
Daniel Dickinson 9fd8e55132 imagebuilder: Fix sorting package list breaks opkg dependency handling for provides
When imagebuild sorts package lists it breaks opkg's ability to realize
that a providers for a Provides has already been installed, when the sort
results in the provider being later in the list of packages that a package
which depends on a Provides (and hence the provider is not yet installed
for opkg to realize the provider was available doesn't not handle the case
of a package that is to be installed satisfying a dependency, only one that
is already installed (or which it schedules to be installed, which in the
absence of an installed provider is whichever provider happens to be the
default)

Signed-off-by: Daniel Dickinson <openwrt@daniel.thecshore.com>
8 years ago
Felix Fietkau f4c4d501e4 build: remove profile kernel/build system config override support
It has been unused for years

Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
Felix Fietkau 80f4988da3 target/imagebuilder: fix using new device profiles
Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
Alexander Couzens 009a069ec0 imagebuilder: rename OpenWrt into LEDE
Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
8 years ago
Jo-Philipp Wich b9466382b5 imagebuilder: use correct package directory when bundling kmods and libc
The libc and kernel package files moved since the introduction of shared
packages and the changed output directory layout. This causes the generated
ImageBuilder archive to lack the necessary "libc" and "kernel" meta packages,
leading to opkg install errors later on.

Use the FeedPackageDir macro to figure out the proper source directory to use.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years ago
Jo-Philipp Wich da46d2b228 imagebuilder: fix standalone operation
Fix standalone ImageBuilders after the package layout rework.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years ago
Felix Fietkau 0a0bfc8507 images imagebuilder: Allow to add sanitized extra name
For final output image names allow user to add an
extra string (which is sanitized).  This is particularly
useful with ImageBuilder where you may generate multiple
images from the same base and for the same board,
with different package selections and additional files
(via FILES=).

Signed-off-by: Daniel Dickinson <openwrt@daniel.thecshore.com>

SVN-Revision: 48083
8 years ago
Felix Fietkau d69efa4e15 targets: Use configured distribution name for SDK, IB, and Toolchain tarball names
We allow to configure the version distribution name;
let's also use it for the tarballs (SDK, ImageBuilder,
and SDK).

Signed-off-by: Daniel Dickinson <openwrt@daniel.thecshore.com>

SVN-Revision: 48081
8 years ago
Jo-Philipp Wich fb3fb9b6c1 imagebuilder: store package lists in cache directory
Force opkg to store the downloaded repository indizes into the cache
directory as well, this way the IB can be used in an offline setting
once all required files have been cached.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 46912
9 years ago
Felix Fietkau add0c00c96 imagebuilder: run build prereq checks before building image to set up host commands properly
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 46840
9 years ago
John Crispin 096b3759a9 build: Prevent more gzip timestamps
To improve reproducibility, prevent the inclusion of timestamps
in the gzip header.

Signed-off-by: Reiner Herrmann <reiner@reiner-h.de>

SVN-Revision: 46361
9 years ago
Jo-Philipp Wich 11f05afd26 imagebuilder: create cache directory if needed (#18629)
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 45866
9 years ago
Jo-Philipp Wich e91342e285 IB: make CONFIG_IB_STANDALONE the default
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 45800
9 years ago