Reuse fixed release-ready outputs and one shared build cache
Custom RELEASE_READY_OUTPUT names now map to build/release-ready, and only build/release-ready-phone keeps a separate purpose, so task and version names no longer leave a new folder per run. Both share build/release-ready/home under an exclusive cache lease held through container exit. The release command uses the same mapping. Phone installs take a lock, select only an APK this run built and reuse the default signed path. Nothing prunes existing output.
This commit is contained in:
1 parent
8408d63f84
commit
b8bfb7ff1c
9 files changed
+416
-32
No files matched your search
@@ -2,6 +2,7 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- Repeated local builds reuse storage instead of keeping a new copy for each run.
|
||||
- Linux: install from the signed APT repository and get updates with apt.
|
||||
- Filter Android account-key imports to key files and related backup types.
|
||||
- Practice results show smooth, labeled mood faces with a clear selection.
|
||||
|
||||
+3
-1
@@ -964,7 +964,9 @@ A release names one fixed commit while peers keep working on master:
|
||||
Daochi sync server. Platform release builds remain separate. The existing
|
||||
repository overlay materializes every candidate file and recreates source
|
||||
directories, so later edits, new files and commits cannot enter the run.
|
||||
Outputs stay in `build/release-ready/`.
|
||||
Outputs stay in `build/release-ready/`. Custom `RELEASE_READY_OUTPUT` names
|
||||
reuse that folder; only `build/release-ready-phone/` has a separate purpose.
|
||||
Both use the shared, leased cache described in [building](building.md).
|
||||
3. Build the Moto APK in the same fixed-source overlay and save its content
|
||||
identity and SHA-256 build receipt. Use `ANDROID_VERIFY_INSTANCE=1` for
|
||||
the separate `xyz.waozi.inbe.verification` debug instance when the owner's
|
||||
|
||||
@@ -112,6 +112,31 @@ publisher setup are described in [app packages](apps.md).
|
||||
Native binaries are written to `build/bin/<platform>/`. Release artifacts are
|
||||
written under `build/dist/`.
|
||||
|
||||
Fixed-commit container builds (`make release-ready` and
|
||||
`scripts/phone-install.sh`) reuse `build/release-ready/` and
|
||||
`build/release-ready-phone/`, respectively. `RELEASE_READY_OUTPUT` still accepts
|
||||
`build/release-ready-NAME`, but names other than `phone` map to
|
||||
`build/release-ready/`; read artifacts and stamps from that canonical path.
|
||||
The release command uses the same mapping. Task names and versions never
|
||||
allocate more output folders. An active output is locked and a second run
|
||||
for it stops before changing its files.
|
||||
|
||||
Both purposes share the existing `build/release-ready/home/`, rather than
|
||||
creating a home per output. Keeping this regenerable cache under `build/` also
|
||||
keeps the fixed-source overlay from traversing it while recreating source
|
||||
directories.
|
||||
Ziran keeps sources by repository URL and exact commit under its `.cache/ziran/`,
|
||||
so different locks reuse identical dependencies while retaining distinct
|
||||
commits. Gradle downloads also reuse this home. Runs hold an exclusive cache
|
||||
lease through container exit, including failed builds; different purposes wait
|
||||
for that lease. Incremental source-time records and compilation outputs stay
|
||||
per purpose. No script prunes existing output or caches, including
|
||||
`promo-video` and `package-releases`. Other old per-output homes are not migrated
|
||||
or deleted automatically. Normal builds and CI keep their existing cache and
|
||||
`build/dist/` paths. Phone signing reuses
|
||||
`build/phone-install/app-ABI-debug-signed.apk` (one of the four supported Android
|
||||
ABIs) even with a versioned `--apk` input, preserving the default signed path.
|
||||
|
||||
Dependencies, including Kryon and the Ziran toolchain, are pinned in
|
||||
`ziran.lock`. `make` fetches them into the Ziran package cache and links them
|
||||
under `build/packages/`; `sh scripts/packages.sh` does the same by hand.
|
||||
|
||||
@@ -84,6 +84,7 @@ phone-check-gate-test:
|
||||
.PHONY: release-tooling-test
|
||||
release-tooling-test:
|
||||
python3 tests/release_test.py
|
||||
python3 tests/build_layout_test.py
|
||||
python3 tests/file_size_test.py
|
||||
|
||||
.PHONY: play-screenshots-test
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate and map bounded outputs for release-ready and the release gate."""
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def release_ready_output(requested="build/release-ready"):
|
||||
if not re.fullmatch(r"build/release-ready(?:-[a-z0-9][a-z0-9-]*)?", requested):
|
||||
raise ValueError("RELEASE_READY_OUTPUT must be build/release-ready or build/release-ready-NAME")
|
||||
if requested == "build/release-ready-phone":
|
||||
return requested
|
||||
return "build/release-ready"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
print(release_ready_output(sys.argv[1]))
|
||||
except ValueError as error:
|
||||
sys.exit(str(error))
|
||||
@@ -27,6 +27,15 @@ keystore=${ANDROID_DEBUG_KEYSTORE:-$HOME/.config/.android/debug.keystore}
|
||||
android_sdk=${ANDROID_SDK_ROOT:-${ANDROID_HOME:-$HOME/Android/Sdk}}
|
||||
output=build/release-ready-phone
|
||||
|
||||
# Serialize the complete build/sign/install operation, including --apk calls.
|
||||
# This lock lives outside reusable output, so no run can replace its inode.
|
||||
mkdir -p .cache/release-ready
|
||||
exec 7>.cache/release-ready/phone-install.lock
|
||||
if ! flock -n 7; then
|
||||
echo "phone-install: another phone install is running" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
device=pixel
|
||||
revision=HEAD
|
||||
apk=""
|
||||
@@ -62,6 +71,10 @@ phone() {
|
||||
}
|
||||
abi=$(phone shell getprop ro.product.cpu.abi | tr -d '\r')
|
||||
[ -n "$abi" ] || fail "could not read the phone's ABI"
|
||||
case "$abi" in
|
||||
arm64-v8a|armeabi-v7a|x86|x86_64) ;;
|
||||
*) fail "unsupported Android ABI: $abi" ;;
|
||||
esac
|
||||
echo "phone-install: $(phone shell getprop ro.product.model | tr -d '\r') ($serial, $abi)"
|
||||
|
||||
if [ "$device" = moto ]; then
|
||||
@@ -75,11 +88,13 @@ fi
|
||||
if [ -z "$apk" ]; then
|
||||
commit=$(git rev-parse "$revision^{commit}")
|
||||
echo "phone-install: building $(git log -1 --format='%h %s' "$commit")"
|
||||
# A failed build must not leave an earlier APK to install.
|
||||
find "$output/build/android" -name "app*-$abi-debug.apk" -delete 2>/dev/null || true
|
||||
# set -e stops a failed build before any earlier APK can be installed.
|
||||
# Select only APKs copied by this run; do not delete another build's files.
|
||||
started=.cache/release-ready/phone-build-start
|
||||
touch "$started"
|
||||
RELEASE_READY_REVISION=$commit RELEASE_READY_OUTPUT=$output RELEASE_READY_INCREMENTAL=1 \
|
||||
sh scripts/release-ready.sh ANDROID_ABI="$abi" android-debug
|
||||
apk=$(find "$output/build/android" -name "app*-$abi-debug.apk" | head -n 1)
|
||||
apk=$(find "$output/build/android" -name "app*-$abi-debug.apk" -newer "$started" | head -n 1)
|
||||
[ -f "$apk" ] || fail "the build left no app-$abi-debug.apk in $output/build/android"
|
||||
fi
|
||||
[ -f "$apk" ] || fail "no APK at $apk"
|
||||
@@ -88,7 +103,7 @@ fi
|
||||
# with this machine's.
|
||||
[ -f "$keystore" ] || fail "no debug keystore at $keystore"
|
||||
build_tools=$(ls -d "$android_sdk"/build-tools/* | sort -V | tail -n 1)
|
||||
signed=build/phone-install/$(basename "$apk" .apk)-signed.apk
|
||||
signed=build/phone-install/app-$abi-debug-signed.apk
|
||||
mkdir -p build/phone-install
|
||||
"$build_tools/zipalign" -f -p 4 "$apk" "$signed.aligned"
|
||||
"$build_tools/apksigner" sign --ks "$keystore" --ks-pass pass:android \
|
||||
|
||||
+45
-21
@@ -10,7 +10,7 @@
|
||||
# edits and new files in the canonical checkout cannot enter the tests. Peers
|
||||
# keep working on master; nothing this container writes reaches their checkout.
|
||||
# Packages use the fixed lock without local overrides or untracked secrets.
|
||||
# Build outputs stay in build/release-ready/ (or RELEASE_READY_OUTPUT); a run
|
||||
# Build outputs reuse build/release-ready/ or build/release-ready-phone/; a run
|
||||
# takes about 90 minutes.
|
||||
#
|
||||
# Covered: ci.yml Secret Scan, Package URLs, large-file guard, all three test
|
||||
@@ -30,16 +30,46 @@ set -eu
|
||||
root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
||||
cd "$root"
|
||||
|
||||
# RELEASE_READY_OUTPUT gives a second run its own build/release-ready-NAME
|
||||
# outputs, so a fixed-commit build can run beside an active check.
|
||||
output=${RELEASE_READY_OUTPUT:-build/release-ready}
|
||||
case "$output" in
|
||||
build/release-ready|build/release-ready-[a-z0-9]*) ;;
|
||||
*) echo "release-ready: RELEASE_READY_OUTPUT must be build/release-ready-NAME" >&2; exit 2 ;;
|
||||
esac
|
||||
case "$output" in
|
||||
*[!a-z0-9/-]*) echo "release-ready: RELEASE_READY_OUTPUT must be build/release-ready-NAME" >&2; exit 2 ;;
|
||||
esac
|
||||
# Legacy task/version names are aliases, never additional permanent outputs.
|
||||
requested_output=${RELEASE_READY_OUTPUT:-build/release-ready}
|
||||
output=$(python3 scripts/check-build-layout.py "$requested_output") || exit 2
|
||||
if [ "$output" != "$requested_output" ]; then
|
||||
echo "release-ready: $requested_output reuses $output"
|
||||
fi
|
||||
|
||||
# Lock before writing any output. Never follow an output symlink into another
|
||||
# purpose's files. There is deliberately no scan or cleanup of old folders.
|
||||
for directory in build "$output" "$output/build" build/release-ready build/release-ready/home; do
|
||||
if [ -L "$directory" ]; then
|
||||
echo "release-ready: refusing symlink $directory" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
mkdir -p "$output"
|
||||
if [ -L "$output/lock" ]; then
|
||||
echo "release-ready: refusing symlink $output/lock" >&2
|
||||
exit 2
|
||||
fi
|
||||
exec 9>"$output/lock"
|
||||
if ! flock -n 9; then
|
||||
echo "release-ready: another run is using $output" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Every fixed-commit run uses the same home, including Ziran's URL+commit
|
||||
# source cache and Gradle's downloads. Hold this lease until Podman exits:
|
||||
# package tools may repair sources and build inside their cached checkouts.
|
||||
# Different output purposes can queue, but cannot mutate this cache together.
|
||||
cache="$root/build/release-ready"
|
||||
mkdir -p "$cache"
|
||||
if [ -L "$cache/cache.lock" ]; then
|
||||
echo "release-ready: refusing symlink $cache/cache.lock" >&2
|
||||
exit 2
|
||||
fi
|
||||
exec 8>"$cache/cache.lock"
|
||||
echo "release-ready: waiting for the shared build cache"
|
||||
flock 8
|
||||
mkdir -p "$cache/home" "$output/build"
|
||||
commit=$(git rev-parse "${RELEASE_READY_REVISION:-HEAD}^{commit}")
|
||||
image_tag=$({
|
||||
git show "$commit:.github/local-ci/Containerfile"
|
||||
@@ -126,7 +156,6 @@ else
|
||||
printf "\nrelease-ready: every covered local CI job passed\n"
|
||||
fi'
|
||||
|
||||
mkdir -p "$output/build" "$output/home"
|
||||
sync_server=""
|
||||
sync_mount_source="$root/$output/sync-unused"
|
||||
mkdir -p "$sync_mount_source"
|
||||
@@ -137,13 +166,6 @@ if [ "$#" -eq 0 ]; then
|
||||
git -C "$sync_server" cat-file -e "$sync_commit^{commit}"
|
||||
sync_mount_source=$sync_server
|
||||
fi
|
||||
# One run at a time: runs share build/release-ready/.
|
||||
exec 9>"$output/lock"
|
||||
if ! flock -n 9; then
|
||||
echo "release-ready: another run is using $output" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Containers here need no session keyring. Creating one counts against the
|
||||
# account's kernel key quota (200 keys), which long-lived sessions fill, and
|
||||
# a full quota stops every container from starting.
|
||||
@@ -183,10 +205,12 @@ podman run --rm --pull=never --userns=keep-id \
|
||||
$dns_option \
|
||||
-v "$root:/src:O" \
|
||||
-v "$root/$output/build:/src/build" \
|
||||
-v "$root/$output/home:/home/ubuntu" \
|
||||
-v "$cache/home:/home/ubuntu" \
|
||||
-v "$sync_mount_source:/sync-server:O" \
|
||||
$android_mount \
|
||||
-e HOME=/home/ubuntu -e CI=true -e RELEASE_READY_COMMIT="$commit" \
|
||||
-e HOME=/home/ubuntu -e XDG_CACHE_HOME=/home/ubuntu/.cache \
|
||||
-e GRADLE_USER_HOME=/home/ubuntu/.gradle \
|
||||
-e CI=true -e RELEASE_READY_COMMIT="$commit" \
|
||||
-e RELEASE_READY_INCREMENTAL="$incremental" \
|
||||
-e RELEASE_READY_SNAPSHOT="$snapshot" \
|
||||
-w /src "$image" sh -c "$inside" release-ready "$@" || status=$?
|
||||
|
||||
+7
-6
@@ -35,12 +35,6 @@ import tempfile
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
# RELEASE_READY_OUTPUT names a separate release-ready run's outputs, as in
|
||||
# scripts/release-ready.sh.
|
||||
READY_OUTPUT = os.environ.get("RELEASE_READY_OUTPUT", "build/release-ready")
|
||||
if not re.fullmatch(r"build/release-ready(-[a-z0-9][a-z0-9-]*)?", READY_OUTPUT):
|
||||
raise SystemExit("RELEASE_READY_OUTPUT must be build/release-ready-NAME")
|
||||
READY_STAMP = ROOT / READY_OUTPUT / "stamp.json"
|
||||
SUBJECT = "Release Inner Breeze {version}"
|
||||
# What ./update_version.sh writes; only these may join the release commit.
|
||||
VERSION_FILES = ("droid/app/build.gradle", "src/core/version.h", "src/core/version.zi", "windows/inbe.rc",
|
||||
@@ -57,6 +51,13 @@ def load(name, path):
|
||||
|
||||
phone_check = load("phone_check", "scripts/phone-check.py")
|
||||
gate = load("check_phone_check", "scripts/check-phone-check.py")
|
||||
build_layout = load("build_layout", "scripts/check-build-layout.py")
|
||||
try:
|
||||
READY_OUTPUT = build_layout.release_ready_output(
|
||||
os.environ.get("RELEASE_READY_OUTPUT", "build/release-ready"))
|
||||
except ValueError as error:
|
||||
raise SystemExit(str(error)) from error
|
||||
READY_STAMP = ROOT / READY_OUTPUT / "stamp.json"
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
"""Exercise the real build wrappers with a fake container, without a checkout or device."""
|
||||
import importlib.util
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
spec = importlib.util.spec_from_file_location("build_layout", ROOT / "scripts/check-build-layout.py")
|
||||
layout = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(layout)
|
||||
|
||||
|
||||
class BuildLayoutTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
# Only fixtures and fake output are removed, never the project's build/.
|
||||
parent = ROOT / "backup/build-layout-tests"
|
||||
parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = tempfile.TemporaryDirectory(dir=parent)
|
||||
self.addCleanup(temporary.cleanup)
|
||||
self.root = Path(temporary.name)
|
||||
(self.root / "scripts").mkdir()
|
||||
(self.root / "bin").mkdir()
|
||||
(self.root / "server").mkdir()
|
||||
for name in ("release-ready.sh", "check-build-layout.py", "phone-install.sh"):
|
||||
shutil.copyfile(ROOT / "scripts" / name, self.root / "scripts" / name)
|
||||
(self.root / "scripts/phone-check.py").write_text(
|
||||
"ROOT = None\ndef release_content(root, commit):\n return commit\n")
|
||||
(self.root / "scripts/check-release-overlay.py").write_text("# snapshot fixture\n")
|
||||
self.executable("git", '''#!/usr/bin/env python3
|
||||
import sys
|
||||
args = sys.argv[1:]
|
||||
if args[0] == '-C':
|
||||
args = args[2:]
|
||||
if args[0] == 'rev-parse':
|
||||
print(args[1].split('^')[0])
|
||||
elif args[0] == 'show':
|
||||
print('fixed image inputs')
|
||||
elif args[0] == 'log':
|
||||
print('fixture revision')
|
||||
elif args[0] != 'cat-file':
|
||||
raise SystemExit('unexpected git: ' + repr(args))
|
||||
''')
|
||||
self.executable("podman", '''#!/usr/bin/env python3
|
||||
import hashlib, json, os, pathlib, sys, time
|
||||
args = sys.argv[1:]
|
||||
if args[:2] == ['image', 'exists']:
|
||||
sys.exit(0)
|
||||
assert args[0] == 'run', args
|
||||
mounts = [args[i+1] for i, value in enumerate(args) if value == '-v']
|
||||
home = pathlib.Path(next(value.rsplit(':', 1)[0] for value in mounts
|
||||
if value.endswith(':/home/ubuntu')))
|
||||
output = pathlib.Path(next(value.rsplit(':', 1)[0] for value in mounts
|
||||
if value.endswith(':/src/build')))
|
||||
environment = dict(args[i+1].split('=', 1) for i, value in enumerate(args) if value == '-e')
|
||||
commit = environment['RELEASE_READY_COMMIT']
|
||||
cache = home / '.cache/ziran/sources'
|
||||
cache.mkdir(parents=True, exist_ok=True)
|
||||
# A shared source and a commit-specific source must coexist across runs.
|
||||
shared = cache / 'shared'
|
||||
reuse = shared.exists()
|
||||
shared.write_text('same dependency')
|
||||
identity = hashlib.sha256(('https://example.invalid/package\\n' + commit).encode()).hexdigest()
|
||||
(cache / identity).write_text(commit)
|
||||
record = dict(home=str(home), output=str(output), commit=commit, reuse=reuse,
|
||||
mounts=mounts, environment=environment)
|
||||
with open('runs.jsonl', 'a') as stream:
|
||||
stream.write(json.dumps(record) + '\\n')
|
||||
if os.environ.get('FIXTURE_BLOCK'):
|
||||
pathlib.Path('entered').touch()
|
||||
deadline = time.monotonic() + 15
|
||||
while not pathlib.Path('continue').exists():
|
||||
assert time.monotonic() < deadline, 'fixture lease timed out'
|
||||
time.sleep(.02)
|
||||
if not os.environ.get('FIXTURE_NO_APK'):
|
||||
(output / 'android').mkdir(exist_ok=True)
|
||||
(output / 'android/app-arm64-v8a-debug.apk').write_bytes(b'fixture apk')
|
||||
sys.exit(int(os.environ.get('FIXTURE_STATUS', '0')))
|
||||
''')
|
||||
self.environment = dict(os.environ, PATH=str(self.root / "bin") + ":" + os.environ["PATH"],
|
||||
RELEASE_READY_REVISION="1" * 40,
|
||||
RELEASE_READY_DAOCHI_SOURCE=str(self.root / "server"),
|
||||
PYTHONDONTWRITEBYTECODE="1")
|
||||
for name in ("RELEASE_READY_OUTPUT", "RELEASE_READY_INCREMENTAL", "RELEASE_READY_DNS"):
|
||||
self.environment.pop(name, None)
|
||||
|
||||
def executable(self, name, text):
|
||||
path = self.root / "bin" / name
|
||||
path.write_text(text)
|
||||
path.chmod(0o755)
|
||||
|
||||
def run_build(self, output="build/release-ready", revision=None, status=0, targets=("android-debug",)):
|
||||
environment = dict(self.environment, RELEASE_READY_OUTPUT=output, FIXTURE_STATUS=str(status))
|
||||
if revision:
|
||||
environment["RELEASE_READY_REVISION"] = revision
|
||||
return subprocess.run(["sh", "scripts/release-ready.sh", *targets], cwd=self.root,
|
||||
env=environment, capture_output=True, text=True, timeout=10)
|
||||
|
||||
def records(self):
|
||||
path = self.root / "runs.jsonl"
|
||||
return [json.loads(line) for line in path.read_text().splitlines()] if path.exists() else []
|
||||
|
||||
def test_two_names_and_commits_share_cache_and_one_output(self):
|
||||
for name, revision in (("build/release-ready-61-apk", "1" * 40),
|
||||
("build/release-ready-62b", "2" * 40)):
|
||||
result = self.run_build(name, revision)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse((self.root / name).exists())
|
||||
first, second = self.records()
|
||||
self.assertEqual(first["home"], second["home"])
|
||||
self.assertEqual(first["output"], second["output"])
|
||||
self.assertFalse(first["reuse"])
|
||||
self.assertTrue(second["reuse"])
|
||||
cache = Path(first["home"]) / ".cache/ziran/sources"
|
||||
self.assertEqual(len(list(cache.iterdir())), 3)
|
||||
self.assertEqual({p.name for p in (self.root / "build").iterdir()},
|
||||
{"release-ready"})
|
||||
self.assertEqual(Path(first["home"]), self.root / "build/release-ready/home")
|
||||
self.assertTrue((self.root / "build/release-ready/build/android/app-arm64-v8a-debug.apk").is_file())
|
||||
|
||||
def test_phone_has_stable_output_and_the_same_home(self):
|
||||
self.assertEqual(self.run_build().returncode, 0)
|
||||
result = self.run_build("build/release-ready-phone", "2" * 40)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
first, second = self.records()
|
||||
self.assertNotEqual(first["output"], second["output"])
|
||||
self.assertEqual(first["home"], second["home"])
|
||||
self.assertTrue(second["reuse"])
|
||||
self.assertEqual(second["environment"]["XDG_CACHE_HOME"], "/home/ubuntu/.cache")
|
||||
self.assertEqual(second["environment"]["GRADLE_USER_HOME"], "/home/ubuntu/.gradle")
|
||||
|
||||
def test_full_run_stamp_uses_the_same_mapping_as_the_release_gate(self):
|
||||
alias = "build/release-ready-release61"
|
||||
result = self.run_build(alias, targets=())
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
stamp = self.root / layout.release_ready_output(alias) / "stamp.json"
|
||||
self.assertTrue(json.loads(stamp.read_text())["passed"])
|
||||
self.assertFalse((self.root / alias).exists())
|
||||
environment = dict(self.environment, RELEASE_READY_OUTPUT=alias)
|
||||
script = ("import importlib.util; "
|
||||
"s=importlib.util.spec_from_file_location('release', 'scripts/release.py'); "
|
||||
"m=importlib.util.module_from_spec(s); s.loader.exec_module(m); print(m.READY_OUTPUT)")
|
||||
result = subprocess.run([sys.executable, "-c", script], cwd=ROOT, env=environment,
|
||||
capture_output=True, text=True, timeout=10)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(result.stdout.strip(), "build/release-ready")
|
||||
|
||||
def start_blocked(self):
|
||||
process = subprocess.Popen(["sh", "scripts/release-ready.sh", "android-debug"], cwd=self.root,
|
||||
env=dict(self.environment, FIXTURE_BLOCK="1"),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
self.addCleanup(self.finish_process, process)
|
||||
self.wait_for(lambda: (self.root / "entered").exists())
|
||||
return process
|
||||
|
||||
def finish_process(self, process):
|
||||
(self.root / "continue").touch()
|
||||
process.communicate(timeout=20)
|
||||
|
||||
def wait_for(self, condition):
|
||||
deadline = time.monotonic() + 5
|
||||
while not condition():
|
||||
self.assertLess(time.monotonic(), deadline, "fixture did not reach expected state")
|
||||
time.sleep(.02)
|
||||
|
||||
def test_active_output_is_not_changed_and_cache_users_are_serialized(self):
|
||||
first = self.start_blocked()
|
||||
with (self.root / "build/release-ready/cache.lock").open("a") as lease:
|
||||
with self.assertRaises(BlockingIOError):
|
||||
fcntl.flock(lease, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
stamp = self.root / "build/release-ready/stamp.json"
|
||||
stamp.write_text("active run's file")
|
||||
result = self.run_build("build/release-ready-another-task")
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("another run is using", result.stderr)
|
||||
self.assertEqual(stamp.read_text(), "active run's file")
|
||||
second = subprocess.Popen(["sh", "scripts/release-ready.sh", "android-debug"], cwd=self.root,
|
||||
env=dict(self.environment, RELEASE_READY_OUTPUT="build/release-ready-phone"),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
self.addCleanup(self.finish_process, second)
|
||||
# The second process owns its output lease before waiting on the cache.
|
||||
self.wait_for(lambda: (self.root / "build/release-ready-phone/lock").exists())
|
||||
self.assertIsNone(second.poll())
|
||||
self.assertEqual(len(self.records()), 1)
|
||||
(self.root / "continue").touch()
|
||||
for process in (first, second):
|
||||
stdout, stderr = process.communicate(timeout=10)
|
||||
self.assertEqual(process.returncode, 0, stdout + stderr)
|
||||
self.assertEqual(len(self.records()), 2)
|
||||
self.assertTrue(self.records()[1]["reuse"])
|
||||
|
||||
def test_failed_build_releases_leases_and_invalidates_old_stamp(self):
|
||||
self.assertEqual(self.run_build(targets=()).returncode, 0)
|
||||
result = self.run_build(status=7)
|
||||
self.assertEqual(result.returncode, 7)
|
||||
self.assertFalse((self.root / "build/release-ready/stamp.json").exists())
|
||||
self.assertEqual(self.run_build().returncode, 0)
|
||||
|
||||
def test_unrelated_outputs_and_old_folders_are_never_pruned(self):
|
||||
paths = ("build/promo-video", "build/package-releases", "build/dist", "build/release-ready-old")
|
||||
for name in paths:
|
||||
path = self.root / name
|
||||
path.mkdir(parents=True)
|
||||
(path / "keep").write_text("owner data")
|
||||
self.assertEqual(self.run_build().returncode, 0)
|
||||
for name in paths:
|
||||
self.assertEqual((self.root / name / "keep").read_text(), "owner data")
|
||||
for name in ("build/promo-video", "build/package-releases", "build/dist",
|
||||
"build/release-ready-../promo-video", "build/release-ready-foo/bar"):
|
||||
result = self.run_build(name)
|
||||
self.assertEqual(result.returncode, 2, result.stderr)
|
||||
|
||||
def test_symlink_output_is_refused_without_touching_target(self):
|
||||
target = self.root / "build/promo-video"
|
||||
target.mkdir(parents=True)
|
||||
(target / "keep").write_text("owner data")
|
||||
(self.root / "build/release-ready").symlink_to(target, target_is_directory=True)
|
||||
result = self.run_build()
|
||||
self.assertEqual(result.returncode, 2)
|
||||
self.assertEqual(list(target.iterdir()), [target / "keep"])
|
||||
self.assertEqual(self.records(), [])
|
||||
|
||||
def test_phone_default_build_and_versioned_apks_reuse_signing_output(self):
|
||||
# Only these fixture executables are on the phone script's PATH.
|
||||
self.executable("adb", '''#!/usr/bin/env python3
|
||||
import hashlib, pathlib, sys
|
||||
args = sys.argv[3:]
|
||||
if args == ['get-state']:
|
||||
print('device')
|
||||
elif args == ['shell', 'getprop', 'ro.product.cpu.abi']:
|
||||
print('arm64-v8a')
|
||||
elif args == ['shell', 'getprop', 'ro.product.model']:
|
||||
print('Fixture')
|
||||
elif args[:3] == ['shell', 'pm', 'path']:
|
||||
print('package:/fixture/base.apk')
|
||||
elif args[:2] == ['shell', 'sha256sum']:
|
||||
data = pathlib.Path('build/phone-install/app-arm64-v8a-debug-signed.apk').read_bytes()
|
||||
print(hashlib.sha256(data).hexdigest(), '/fixture/base.apk')
|
||||
elif args[:2] not in (['install', '-r'], ['shell', 'am'], ['shell', 'pidof']):
|
||||
raise SystemExit('unexpected adb: ' + repr(args))
|
||||
''')
|
||||
sdk = self.root / "sdk/build-tools/1"
|
||||
sdk.mkdir(parents=True)
|
||||
for name, source in {
|
||||
"zipalign": "import shutil, sys; shutil.copyfile(sys.argv[-2], sys.argv[-1])",
|
||||
"apksigner": ("import shutil, sys; "
|
||||
"sys.exit(0) if sys.argv[1] == 'verify' else "
|
||||
"shutil.copyfile(sys.argv[-1], sys.argv[sys.argv.index('--out')+1])"),
|
||||
}.items():
|
||||
path = sdk / name
|
||||
path.write_text("#!/usr/bin/env python3\n" + source + "\n")
|
||||
path.chmod(0o755)
|
||||
key = self.root / "debug.keystore"
|
||||
key.touch()
|
||||
environment = dict(self.environment, ANDROID_SDK_ROOT=str(self.root / "sdk"),
|
||||
ANDROID_DEBUG_KEYSTORE=str(key))
|
||||
|
||||
def install(*arguments):
|
||||
result = subprocess.run(["sh", "scripts/phone-install.sh", "--device", "fixture", *arguments],
|
||||
cwd=self.root, env=environment, capture_output=True, text=True, timeout=10)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
|
||||
install()
|
||||
record = self.records()[0]
|
||||
self.assertEqual(Path(record["output"]), self.root / "build/release-ready-phone/build")
|
||||
for version in ("61", "62"):
|
||||
apk = self.root / ("inbe-" + version + ".apk")
|
||||
apk.write_bytes(version.encode())
|
||||
install("--apk", str(apk))
|
||||
self.assertEqual([p.name for p in (self.root / "build/phone-install").iterdir()],
|
||||
["app-arm64-v8a-debug-signed.apk"])
|
||||
signed = self.root / "build/phone-install/app-arm64-v8a-debug-signed.apk"
|
||||
self.assertEqual(signed.read_bytes(), b"62")
|
||||
failed = subprocess.run(["sh", "scripts/phone-install.sh", "--device", "fixture"],
|
||||
cwd=self.root, env=dict(environment, FIXTURE_STATUS="9"),
|
||||
capture_output=True, text=True, timeout=10)
|
||||
self.assertEqual(failed.returncode, 9, failed.stdout + failed.stderr)
|
||||
self.assertNotIn("phone-install: installed", failed.stdout)
|
||||
self.assertEqual(signed.read_bytes(), b"62")
|
||||
stale = subprocess.run(["sh", "scripts/phone-install.sh", "--device", "fixture"],
|
||||
cwd=self.root, env=dict(environment, FIXTURE_NO_APK="1"),
|
||||
capture_output=True, text=True, timeout=10)
|
||||
self.assertEqual(stale.returncode, 1, stale.stdout + stale.stderr)
|
||||
self.assertIn("the build left no", stale.stderr)
|
||||
self.assertNotIn("phone-install: installed", stale.stdout)
|
||||
self.assertEqual(signed.read_bytes(), b"62")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user