Commit Graph

122 Commits (4a6795409d1520fd3da3e909a8bcf9d7fd0927bb)

Author SHA1 Message Date
Roman Yeryomin 2ca084ccaa build: improve ccache support
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.

Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target

Signed-off-by: Roman Yeryomin <roman@advem.lv>
4 years ago
Paul Spooren 07449f692c build: refactor JSON info files to `profiles.json`
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").

They are useful for firmware wizards and script checking for
reproducibility.

Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.

This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.

Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.

Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.

As before, this creation is enabled by default only if `BUILDBOT` is set.

Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
4 years ago
Kevin Darbyshire-Bryant 0c4d91f121 build: clean menuconfig utility as part of dirclean
When sharing a common build directory between different build platforms
eg. macos v docker based linux v virtual machine, a 'make dirclean'
isn't quite enough to clean all the platform related binaries.  The
'conf' and 'mconf' aka 'make menuconfig/defconfig & friends' utilities
remain.

Clean those as part of 'dirclean' so they get rebuilt for the current
platform on the next 'make'

Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
4 years ago
Paul Spooren 6caf437652 build: add buildinfo as single Makefile target
The prepare target was added some 11 years ago to build tools and
toolchain and was recently extended to  create buildinfo files for
reproducibility, meaning {feeds,version,config}.buildinfo.

As the buildbot workflow is more complex than the single prepare (kmod
feed insertion), prepare is only used to create those buildinfo files.
Running prepare however runs `target/compile` as well, taking time even
everything is already compiled.

Splitting this allows the buildbot to run only the `buildinfo` target
while others can still use the convenience feature `prepare`.

Signed-off-by: Paul Spooren <mail@aparcar.org>
5 years ago
Petr Štetiar 965f341aa9 build: fix host menu config targets using ncurses
On a recent Gentoo Linux installation, invoking `make menuconfig`, `make
kernel_menuconfig` or `make kernel_nconfig` in the build system fails,
whereas for example `make menuconfig` in the kernel tree alone works as
expected.

This is happening because STAGING_PREFIX is not defined when kernel's
{menu,n}config target calls pkg-config from the toolchain/host and thus
pkg-config returns an empty value, and the fallback values in the kernel
config script are applied but those are off and the linking fails.

Solution is to use system's pkg-config for all ncurses based menu config
targets in order to provide proper compiler/linker flags.

Ref: FS#2423
Cc: Thomas Albers <thomas.gameiro@gmail.com>
Tested-by: Stijn Tintel <stijn@linux-ipv6.be>
Tested-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 years ago
Paul Spooren 454021581f build: add buildinfo files for reproducibility
generate feeds.buildinfo and version.buildinfo in build dir after
containing the feed revisions (via ./scripts/feeds list -sf) as well as
the current revision of buildroot (via ./scripts/getver.sh).

With this information it should be possible to reproduce any build,
especially the release builds.

Usage would be to move feeds.buildinfo to feeds.conf and git checkout the
revision hash of version.buildinfo.

Content of feeds.buildinfo would look similar to this:

    src-git routing https://git.openwrt.org/feed/routing.git^bf475d6
    src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e
    ...

Content of version.buildinfo would look similar to this:

    r10203+1-c12bd3a21b

Without the exact feed revision it is not possible to determine
installed package versions.

Also rename config.seed to config.buildinfo to follow the recommended
style of https://reproducible-builds.org/docs/recording/

Signed-off-by: Paul Spooren <mail@aparcar.org>
5 years ago
Felix Fietkau 7ec092e641 Revert faulty tree push
Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85)
Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e)
Revert "ramips: add preliminary support for WIO ONE" (085141dc5b)
Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539)
Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d)
Revert "toolchain: add autoconf-lean" (fdb30eed03)
Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758)

Signed-off-by: Felix Fietkau <nbd@nbd.name>
5 years ago
Felix Fietkau 363ce4329d build: use config.site generated by autoconf-lean, drop hardcoded sitefiles
Signed-off-by: Felix Fietkau <nbd@nbd.name>
5 years ago
Petr Štetiar 12610a5620 Revert "build: allow simple build system customization with local.mk"
This reverts commit fe928c7a81 as it seems
to introduce build breakage when local.mk doesn't exist.

 make --trace
 toplevel.mk:216: target 'local.mk' does not exist

Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 years ago
Petr Štetiar fe928c7a81 build: allow simple build system customization with local.mk
I tend to automate most of the boring and repetitive tasks like firmware
flashing, build config reconfiguration etc., so I always end up adding
`include local.mk` as a last line in my Makefile, where local.mk is
usually symlink to some other place, just to not accidentally delete it
during `git clean`.

Carrying this single uncommited modification along in the development
process is quite PITA, because it's causing problems during Git
workflow, while rebasing etc.

I hope, that I'm not alone using similar workflow, so I believe, that
this modification might be useful for others as well.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 years ago
Thomas Langer 41faf52b0f build: Unset CDPATH to avoid problems
In some places the output of commands, which include "cd" are used.
In case of CDPATH the new path is printed, which might not be expected.
Disable the variable to avoid these problem.

When CDPATH was set by the user to some value like "export CDPATH=."
the git checkout done by the build system did not work anymore, the
git cloning aborted with such an error message for example:
....
Packing checkout...
tar: /disk/fs1/tmp2/mehrtens/pon-ugw/ugw-haps/openwrt/tmp/dl/ppa-drv-1.0\n@1534240258: Cannot stat: No such file or directory
tar: Date sample file not found
Try 'tar --help' or 'tar --usage' for more information.
.....

To avoid this, this patch makes the build system unset CDPATH inside
the build system, so the build system will still work even when the
user set this variable in his local environment.

Signed-off-by: Thomas Langer <thomas.langer@intel.com>
Signed-off-by: Hauke Mehrtens <hauke.mehrtens@intel.com>
Acked-by: Hans Dedecker <dedeckeh@gmail.com>
6 years ago
Jo-Philipp Wich 681e825f8f build: include package directory in sha256sums when running on buildbot
In order to be able to better compare files to sync in the future, include
all BIN_DIR subdirectories in the checksum calculation.

To not break existing applications, restrict the recursive checksumming to
CONFIG_BUILDBOT for now.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
6 years ago
Zoltan HERPAI 34bbbbf9c3 merge: base: update base-files and basic config
Signed-off-by: Zoltan HERPAI <wigyori@uid0.hu>
6 years ago
Vitaly Chekryzhev 3ceb0fa7c5 build: prepare config.seed before package compilation
Signed-off-by: Vitaly Chekryzhev <13hakta@gmail.com>
7 years ago
Felix Fietkau 9467ce42da build: get rid of host.mk
Defined required host related variables in toplevel.mk instead

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Felix Fietkau 664918d891 Revert "build: always run package/cleanup before package/compile"
This reverts commit 2990a21058.
This introduces a race condition, let's fix this in buildbot instead.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Felix Fietkau 2990a21058 build: always run package/cleanup before package/compile
Remove unnecessary stampfile indirection
Fixes an issue with the command sequence used by buildbot

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Felix Fietkau 131db366a3 build: remove separate /install step for host builds
Reduces the number of recursive make invocations

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Matthias Schiffer 3d71786154
build: move STAGING_DIR_HOSTPKG and BUILD_DIR_HOST back to a common directory for all targets
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.

$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
7 years ago
Hannu Nyman c1b12aa838 Makefile: ensure that BIN_DIR exists for diffconfig
Ensure that BIN_DIR exists when the diffconfig target needs it.
Otherwise 'make diffconfig' fails after 'make clean'

Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
7 years ago
Jo-Philipp Wich 2a72a916ab build: add diffconfig target
Add a "diffconfig" build target which stores the output of
"scripts/diffconfig.sh" as "config.seed" in the image output directory and
invoke that target by default.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years ago
Felix Fietkau 7a315b0b5d build: implement make check and make package/X/check
This is intended to be used for a wide array of package sanity checks.

The first check that is implemented is for the hash of downloaded files.
It checks:
  - Missing hash
  - Use of SHA256 instead of MD5
  - dl/<file> hash not matching hash in makefile
  - deprecated MD5SUM variable

The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years ago
Felix Fietkau f2de33b2c9 build: move merged package directory from bin/ to staging_dir
Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
Jo-Philipp Wich 27854a0a84 build: add checksum target
Add a new "checksum" make target which generates an sha256sums file over the
image files produced in bin/targets/ and automatically call it during make
world after the package index generation.

The advantage of this new target is that it is guaranteed to run after the
images, the SDK and the ImageBuilder archives have been generated to ensure
that they all end up in the checksum file. Fixes FS#51.

Uses sed to postprocess the OpenSSL digest output into an sha256sum command
compatible format.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years ago
Felix Fietkau 30d955a7a5 build: fix make clean, delete package directories for selected arch
Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years ago
John Crispin fa69553900 branding: add LEDE branding
Signed-off-by: John Crispin <blogic@openwrt.org>
8 years ago
Felix Fietkau 868d5f9ae9 Makefile: set timezone to UTC
This is necessary for reproducible image builds.

Signed-off-by: bryan newbold <bnewbold@robocracy.org>

SVN-Revision: 48538
8 years ago
Felix Fietkau b20a2b4890 Makefile: move the cleaning of staging_dir/target* from dirclean to clean
Currently "make clean" only clears the build_dir/target*, but leaves
staging_dir/target* intact. "make clean" should also clean the
staging_dir/target* directories, as in the current situation some old
packages or libraries may be linked into the firmware from staging_dir
despite a "make clean".

The patch reorganises clean / dirclean functionality so that
* "make clean" also clears the staging_dir/target* in addition to
build_dir/target*.
* "make dirclean" clears toolchain and host(=tools) directories from both
build_dir and staging_dir

signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>

SVN-Revision: 45973
9 years ago
John Crispin 9a96820d3d Makefile: remove non-existent STAGING_DIR_TOOLCHAIN from dirclean
Makefile: remove non-existent STAGING_DIR_TOOLCHAIN from dirclean

Openwrt's top level Makefile uses STAGING_DIR_TOOLCHAIN in the make dirclean
statement.
https://dev.openwrt.org/browser/trunk/Makefile#L55
    rm -rf $(STAGING_DIR) $(STAGING_DIR_HOST) $(STAGING_DIR_TOOLCHAIN)
$(TOOLCHAIN_DIR) $(BUILD_DIR_HOST) $(BUILD_DIR_TOOLCHAIN)

As far as I can determine, no such variable has been defined. I made a search
in Openwrt source repository and the one line in Makefile's dirclean command
is the only place where that variable exists.

The item has been introduced to Makefile by r8362, but even at that time
neither Makefile nor rules.mk defined such a variable. Most likely the goal
has been to set both staging_dir/toolchain and build_dir/toolchain to be
cleaned, but one of the variables has been erroneous. The correct variable
for build_dir/toolchain has been then added by r13494.

References:
https://dev.openwrt.org/changeset/8362/
https://dev.openwrt.org/browser/trunk/Makefile?rev=8362
https://dev.openwrt.org/browser/trunk/rules.mk?rev=8362
https://lists.openwrt.org/pipermail/openwrt-devel/2007-August/001159.html
https://dev.openwrt.org/changeset/13494

In current code,
    TOOLCHAIN_DIR = $(TOPDIR)/staging_dir/$(TOOLCHAIN_DIR_NAME)
    BUILD_DIR_TOOLCHAIN = $(TOPDIR)/build_dir/$(TOOLCHAIN_DIR_NAME)
so the item STAGING_DIR_TOOLCHAIN in the rm command is unnecessary.

signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>

SVN-Revision: 45736
9 years ago
Felix Fietkau 5868331214 build: don't call prereq for any package/symlinks rules
Most of the time, we want to make sure OpenWrt has been configured and
setup before start running make. However, in case of package/symlinks,
forcing prereq as a dependency creates multiple issues:
*when executed on a clean workspace, it will prompt for user input
 and open a menuconfig window before executing the feeds command
*the only way around that is to provide a .config. However, the "prereq"
 target would then run a "make defconfig", which will remove all the
 packages in the .config but from external feeds, as feeds have not been
 installed yet.

The only way to currently work around this, is to generate a fake config
by running "make defconfig", then "make package/symlinks", copy the real
config (which at this point disregards the previously generated config),
and run make defconfig again. Something like this:

make defconfig
make package/symlinks
cp real.config .config
make defconfig

This change is removing the need for the first defconfig, making the
process more logical for OpenWrt users using the package/symlinks target.

Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>

SVN-Revision: 45657
9 years ago
Felix Fietkau 74ca7e9af3 build: clean up sitefiles, make them per-arch
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 45090
9 years ago
Felix Fietkau 4fde9deed1 Revert "build: use ONESHELL to speed up scanning and the toplevel makefile"
This seems to cause issues on some build hosts, and it is not that
important.

SVN-Revision: 43025
10 years ago
Felix Fietkau c497990834 build: use ONESHELL to speed up scanning and the toplevel makefile
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 43021
10 years ago
Felix Fietkau bb959c0f3e build: remove duplicate variable definitions
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 39981
10 years ago
Felix Fietkau c233a088f0 build: it is no longer necessary to call make target/linux/clean as part of make clean
SVN-Revision: 34451
11 years ago
Felix Fietkau 43f34a93e6 Makefile: add a check to bail out early when the path to the openwrt directory contains spaces (#12344)
SVN-Revision: 33884
12 years ago
Felix Fietkau 0f6e5b76f8 build: add back the package/cleanup step to remove the root staging dir for mklibs
SVN-Revision: 33777
12 years ago
Felix Fietkau 44cd20b802 build: rework the package/install step - collect package install lists during package/compile, then install all packages at once
SVN-Revision: 33631
12 years ago
Jo-Philipp Wich b81354d828 abort built in prereq target if there is no site config file for the current target
SVN-Revision: 29798
12 years ago
Felix Fietkau 00d1e599d7 add a command for printing a cleaned up make target database - will be used to analyze package dependencies at some point
SVN-Revision: 22871
14 years ago
Florian Fainelli 7fc192e9a5 fix for GREP_OPTIONS
When using GREP_OPTIONS to supply default options to grep, the buildsystem might get broken (For example adding --color=always breaks it)
This patch will empty the GREP_OPTIONS to prevent the described (and any other) problems related to GREP_OPTIONS

Signed-off-by: Maarten Bezemer <m.m.bezemer@utwente.nl>

SVN-Revision: 22443
14 years ago
Felix Fietkau 61f503c77a add a "make prepare" target which builds everything up to target/compile, useful for scripts/deptest.sh
SVN-Revision: 16153
15 years ago
Ralph Hempel c4d7af7094 added BUILD_LOG_DIR variable, make clean will delete the ./logs dir
SVN-Revision: 16148
15 years ago
Felix Fietkau 882190b198 avoid implicit rules in even more places
SVN-Revision: 14737
15 years ago
Nicolas Thill 71e8b3c4f7 remove toolchain build directory as well in 'make dirclean'
SVN-Revision: 13494
16 years ago
Felix Fietkau a3a37d0bb1 rework parallel building to get rid of some warnings and add back support for parallelizing the kernel build fixes #3882
SVN-Revision: 12322
16 years ago
Felix Fietkau 1b838dbd95 make package prereq checks behave more like build prereq checks (first check all, then fail if necessary), also make them less verbose
SVN-Revision: 12106
16 years ago
Felix Fietkau 4c665e6ca4 large improvement for parallel builds. works without V=99 now and without warnings. tested with -j on an 2x dual core opteron machine
SVN-Revision: 11414
16 years ago
Ralph Hempel 585997f8d2 fix package/symlinks target, will call scripts/feeds update / install introduce new package/symlinks-install and package/symlinks-clean targets - package/symlink-install will re-create the index files and the symlinks for all ./feeds (no update from repositories is performed) - package/symlink-clean will delete all installed symlinks (keeping the ./feeds untouched)
SVN-Revision: 10634
16 years ago
Felix Fietkau 18bd5286e6 use scripts/feeds instead of scripts/feeds.sh for package/symlinks
SVN-Revision: 9990
16 years ago