-
Notifications
You must be signed in to change notification settings - Fork 134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reproducible Builds #539
Comments
I believe i built the apk before posting the commit. So that could be the case. I have some changes pending. I'll commit those and build an apk for you to compare. |
Which is one way to guarantee it's not RB, yeah – see the linked hints, §1 😜
Will that be a new release then? Because only releases are relevant. I could of course test-run it first to see if it is RB – just in case there are other culprits, to ensure we cover them all and have a fine RB release then. For that just ping me with an APK and the information which commit it was built from. Thanks! |
Alright, next release it is. Will let you know |
Try this apk using the latest commit |
Could you please attach the APK here (simply rename it to |
Cant send it here because the max limit is 25mb. What other method do you prefer |
Maybe temporarily attach it to the release (with a "test name"), I'll let you know once I've grabbed it so you can remove it again? Don't forget to name the commit hash I should check against. |
ok |
Build failed:
Are those dependencies in some private repo(s)? |
No |
Well, JCenter closed a couple of weeks ago – and that's where NewPipeExtractor seems to be available at Jitpack, which you have enabled, so I wonder why that is not found. Can you check, please – and fix those dependencies which no longer resolve? |
I'll have a look at it later today |
I switched to exomedia 5.1.0 and that built fine What error is popping up? |
The one shown above. If you give me a commit hash and its matching APK, I can re-check if you want. |
@IzzySoft the test release was somewhat of a disaster. People still downloaded it and threw me errors 😁. Need to find another way to give you the apk. Like there isnt any other file hosting service i can use? |
Guess if you name the release "THIS WILL EAT YOUR CAT!!!" they'd still try it out, huh? 🙈 Are you using Github Actions to build? I see there's a workflow defined. If so, you could link me to the artifact maybe. As for file hosting services: haven't use any in ages, so I cannot tell. The one you used last time wanted me to agree to their TOS and all, and that's a bit much for just downloading a test file… |
Ppl use obtainium and that will grab new releases no matter what he writes in the release or filename |
that cannot be helped then. Can Obtainium not be told to ignore pre-releases? Oof… |
@IzzySoft i managed to get the github action build to succeed. I just now need to find a way to create apks right in github. I guess that would be the best course of action. |
Sounds good! 🤞 Doing it via Github CI ensures builds from a clean tree. Dirty trees are the main reason of RB fails, so this sounds good 😃 |
@IzzySoft the latest action build generated the apks you can check out and see. Also about that newpipe error, somehow gradle wanted the company name to be in lowercase to work. 😁 |
Now I'll need to figure how to find that artifact and how to match it to which commit it was built from… OK, the latter is clear. The former not: there's only a 250+ MB ZIP file… Eek, and not even downloadable via OK, managed to download the full ZIP, extract the APK, and running the RB check. But it's not RB: -rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 16e72a9e META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 5125 b- 5125 stor 1981-01-01 01:01:02 e6456266 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 5125 b- 5125 stor 1981-01-01 01:01:02 0f8bf6e7 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 396 b- 396 stor 1981-01-01 01:01:02 48b4a4cc assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 10043204 b- 4354747 defN 1981-01-01 01:01:02 f3c4df5a classes.dex
+ -rw-r--r-- 0.0 unx 10042968 b- 4354666 defN 1981-01-01 01:01:02 6fd96133 classes.dex
-rw-r--r-- 0.0 unx 631400 b- 254102 defN 1981-01-01 01:01:02 52f51e23 classes2.dex
-rw-r--r-- 0.0 unx 2362568 b- 852191 defN 1981-01-01 01:01:02 2a243272 lib/arm64-v8a/libaria2c.so
-rw-r--r-- 0.0 unx 1672335 b- 1665102 defN 1981-01-01 01:01:02 648a6fac lib/arm64-v8a/libaria2c.zip.so
@@ -1667,7 +1667,7 @@
-rw---- 0.0 fat 884 b- 884 stor 1981-01-01 01:01:02 0ef1b92c res/zz.png
-rw---- 0.0 fat 532 b- 205 defN 1981-01-01 01:01:02 a50d3889 res/zz.xml
-rw---- 0.0 fat 2922860 b- 2922860 stor 1981-01-01 01:01:02 7f51ca9f resources.arsc
- -rw-r--r-- 0.0 unx 150110 b- 68236 defN 1981-01-01 01:01:02 b3cdd6b7 META-INF/CERT.SF
- -rw-r--r-- 0.0 unx 1375 b- 1102 defN 1981-01-01 01:01:02 e1a8e130 META-INF/CERT.RSA
- -rw-r--r-- 0.0 unx 150036 b- 64662 defN 1981-01-01 01:01:02 413b0dcc META-INF/MANIFEST.MF
- 1670 files, 46855332 bytes uncompressed, 37623193 bytes compressed: 19.7%
+ -rw-r--r-- 0.0 unx 150110 b- 68235 defN 1981-01-01 01:01:02 621c0082 META-INF/CERT.SF
+ -rw-r--r-- 0.0 unx 1167 b- 1021 defN 1981-01-01 01:01:02 be28c0d6 META-INF/CERT.RSA
+ -rw-r--r-- 0.0 unx 150036 b- 64657 defN 1981-01-01 01:01:02 0d450b76 META-INF/MANIFEST.MF Next to the
Hmpf. Guess I'll need to update my recipe and patch out some more signing stuff from - #5 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigkeyAlias'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "S4n1t1z3D"
- #6 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigkeyPassword'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "S4n1t1z3D"
- #7 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigstoreFile'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "release_keystore.jks"
- #8 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigstorePassword'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "S4n1t1z3D" Your artifact's APK contains the build config's signing data? 👀 😱 Guess it's time for a new certificate then (in the diff, I've sanitized the values except for the keystore file). Was it that way with previous releases as well? (checking) oof, unfortunately yes – so we must consider your signing key compromised, and you need to change to a new one. You could use Signing Key Rotation, but that will only help devices running Android-9 or later (so 6-8 would need an uninstall/reinstall). |
Why new certificate? How can you even figure out the signing data? Am i not supposed to sign the app? |
Or does the github action configuration not build the apk properly? |
Look above in the diff: there's the keystore, the alias and the passwords. What else do I need to properly sign an APK in your name? Do I miss something? Ah, maybe the keystore is just mentioned but luckily was not checked in. (checking) OK, it's generated at build time from a secret, oof. OK, then maybe no need to change the key itself. But the passwords at least, and make sure they're not ending up in the APK anymore then. Luckily at least the keystore itself didn't end up in the APK. So if you consider its storage safe enough (and thus the risk of someone obtaining it low enough), maybe that should be sufficient. And that |
The confusing part is that the passwords are also stored in secrets. But i create the local.properties file in the github actions to build the apk. Did you get the password from that? Did the file not delete after the build? Can you guide me how to resolve this? |
I think buildconfig gets data from local.properties. If i add the passwords to another .properties file ill be good. Testing that |
I get the passwords when using Apktool to unpack it and generate the Smali code from it. They are stored in a class named
Not sure, but I'm no Android dev. I've seen other projects using
OK, will do… BuildConfig is still there, but the sensitive information is gone from it. Now let's see what the verification builder say: not RB. Hm. APK diff: -rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 1aac3f57 META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 5104 b- 5104 stor 1981-01-01 01:01:02 430965b1 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 5105 b- 5105 stor 1981-01-01 01:01:02 232ce725 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 396 b- 396 stor 1981-01-01 01:01:02 baf29ca2 assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 10045520 b- 4355000 defN 1981-01-01 01:01:02 a4800405 classes.dex
+ -rw-r--r-- 0.0 unx 10045520 b- 4354997 defN 1981-01-01 01:01:02 c16457ad classes.dex
-rw-r--r-- 0.0 unx 631400 b- 254102 defN 1981-01-01 01:01:02 52f51e23 classes2.dex
-rw-r--r-- 0.0 unx 2362568 b- 852191 defN 1981-01-01 01:01:02 2a243272 lib/arm64-v8a/libaria2c.so
-rw-r--r-- 0.0 unx 1672335 b- 1665102 defN 1981-01-01 01:01:02 648a6fac lib/arm64-v8a/libaria2c.zip.so
@@ -140,7 +140,7 @@
-rw---- 2.0 fat 25972 b- 7092 defN 1981-01-01 01:01:02 0eee32e3 org/mozilla/javascript/resources/Messages_fr.properties
-rw---- 2.0 fat 2579 b- 1042 defN 1981-01-01 01:01:02 1e1d6243 org/mozilla/javascript/tools/debugger/test.js
-rw---- 2.0 fat 8222 b- 2787 defN 1981-01-01 01:01:02 9bda5daa org/mozilla/javascript/tools/resources/Messages.properties
- -rw---- 0.0 fat 42792 b- 7129 defN 1981-01-01 01:01:02 1d803ecb AndroidManifest.xml
+ -rw---- 0.0 fat 42792 b- 7135 defN 1981-01-01 01:01:02 133a5067 AndroidManifest.xml
-rw---- 0.0 fat 744 b- 344 defN 1981-01-01 01:01:02 1312bbae res/-1.xml OK, still the Dex. But also the Manifest? (Edit: removed this section as it turned out irrelevant) So let's see the dex diff: -checksum : 9ddf47b4
-signature : 63dd...7072
+checksum : ea914968
+signature : aa7b...7cc1
file_size : 10045520
header_size : 112
link_size : 0
@@ -409077,7 +409077,7 @@
name : 'VERSION_CODE'
type : 'I'
access : 0x0019 (PUBLIC STATIC FINAL)
- value : 107090100
+ value : 107090104
#4 : (in Lcom/deniscerri/ytdl/BuildConfig;) Much cleaner. What irritates me there… isn't that the sed -r '/abi \{/,/}/ { s/include.*/include "arm64-v8a"/ ; s/universalApk true/universalApk false/ }' -i app/build.gradle Why does that result in the arm64 APK getting the "base
So no fix definition, but looking for the build output. Let me try to confirm (just disabling the UniversalApk but having it build all defined ABIs)… Ah, that looks better: just the dex diff (and hence baseline.prof) remain now. Eh, and the dex diff is just … the Oof: "upstream_signed_apk_sha256": "9c37a627b88d8a2c6ea62344407f826f1b0fbc7c8ecd4c6bb962ab4582e79289",
"built_unsigned_apk_sha256": "a63e4872d64844b76b1ef9a1e1c00f6588b0a0bb91000c41cf390912a94ac4e8",
"signature_copied_apk_sha256": "9c37a627b88d8a2c6ea62344407f826f1b0fbc7c8ecd4c6bb962ab4582e79289" OK, that's RB now – congrats! Where do we want to go from here? Leave it as-is (building additional 4 APKs we then just throw away) – or try with the mapping? |
Well i thought I'd build everything in github action and use those for release. |
Yeah, and you build everything in one action, so there it totally makes sense. The verification builders for IoD as well as per-ABI builds with F-Droid.org would build each ABI separately. While the recipe in my builder would just cover the ABI present at IoD, F-Droid would have 4 build blocks (one for each ABI) and thus build all 5 variants (4x ABI plus the UniversalApk) 4 times. I'm no Android dev, so it's hard for me to tell how much overhead that might be (as long as it's not pulling additional files for each ABI, it would most likely be some additional local processing concerning the ABI specific stuff). Which is why I raised the question with you. If you say you want to stay with the process as it currently is, I'd add the (now working) recipe to my builder (just adding a note to it for this), and the green shield for confirmed RBs will be established starting with this version – once you've released it. If you say you're willing to give the "mapping" a chance, I happily try another time having the |
element-hq/element-x-android#2911
|
AFAIK Element X won't build reproducibly unless you build multiple ABIs either (because the manifest will differ otherwise). So doing that might be easiest. I suspect it's fixable but the gradle DSL stuff is pretty complex. My guess is there are no ABI filters when there is only one ABI configured and thus the lookup fails. Thus I suspect simply adding a second
|
Same for
v1.7.9.2 without touching the ABIs (i.e. building all of them) is RB. Now with this recipe (i.e. adding the first 2 lines): build:
- sed -r '/abi \{/,/}/ { s/include.*/include "arm64-v8a"/ ; s/universalApk true/universalApk false/ }' -i app/build.gradle
- sed -r 's/abiCodes.get\(.*\)/abiCodes.get("arm64-v8a")/' -i app/build.gradle
- sed -r '/signingConfigs.debug/d' -i app/build.gradle
- chmod +x gradlew
- touch local.properties
- ./gradlew assembleRelease is NOT RB. APK diff: -rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 73f7a028 META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 5106 b- 5106 stor 1981-01-01 01:01:02 505830c3 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 5106 b- 5106 stor 1981-01-01 01:01:02 104ca8c8 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 396 b- 396 stor 1981-01-01 01:01:02 48b4a4cc assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 10043656 b- 4354933 defN 1981-01-01 01:01:02 54462bfd classes.dex
+ -rw-r--r-- 0.0 unx 10043656 b- 4354933 defN 1981-01-01 01:01:02 d87fbd41 classes.dex
-rw-r--r-- 0.0 unx 631400 b- 254102 defN 1981-01-01 01:01:02 52f51e23 classes2.dex
-rw-r--r-- 0.0 unx 2362568 b- 852191 defN 1981-01-01 01:01:02 2a243272 lib/arm64-v8a/libaria2c.so
-rw-r--r-- 0.0 unx 1672335 b- 1665102 defN 1981-01-01 01:01:02 648a6fac lib/arm64-v8a/libaria2c.zip.so
@@ -140,7 +140,7 @@
-rw---- 2.0 fat 25972 b- 7092 defN 1981-01-01 01:01:02 0eee32e3 org/mozilla/javascript/resources/Messages_fr.properties
-rw---- 2.0 fat 2579 b- 1042 defN 1981-01-01 01:01:02 1e1d6243 org/mozilla/javascript/tools/debugger/test.js
-rw---- 2.0 fat 8222 b- 2787 defN 1981-01-01 01:01:02 9bda5daa org/mozilla/javascript/tools/resources/Messages.properties
- -rw---- 0.0 fat 42792 b- 7129 defN 1981-01-01 01:01:02 11b141f1 AndroidManifest.xml
+ -rw---- 0.0 fat 42792 b- 7133 defN 1981-01-01 01:01:02 1f0b2f5d AndroidManifest.xml
-rw---- 0.0 fat 744 b- 344 defN 1981-01-01 01:01:02 9d59dbe0 res/-1.xml Manifest: E: activity-alias (line=68)
- A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100000
- A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.Default" (Raw: "com.deniscerri.ytdl.Default")
- A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=true
+ A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100002
+ A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.LightIcon" (Raw: "com.deniscerri.ytdl.LightIcon")
+ A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=false
A: http://schemas.android.com/apk/res/android:exported(0x01010010)=true
A: http://schemas.android.com/apk/res/android:launchMode(0x0101001d)=2
A: http://schemas.android.com/apk/res/android:configChanges(0x0101001f)=0x00000004
A: http://schemas.android.com/apk/res/android:targetActivity(0x01010202)="com.deniscerri.ytdl.MainActivity" (Raw: "com.deniscerri.ytdl.MainActivity")
A: http://schemas.android.com/apk/res/android:windowSoftInputMode(0x0101022b)=0x00000020
- A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100003
+ A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100005
E: meta-data (line=78)
A: http://schemas.android.com/apk/res/android:name(0x01010003)="android.app.shortcuts" (Raw: "android.app.shortcuts")
A: http://schemas.android.com/apk/res/android:resource(0x01010025)=@0x7f170008
@@ -127,15 +127,15 @@
E: data (line=119)
A: http://schemas.android.com/apk/res/android:mimeType(0x01010026)="application/txt" (Raw: "application/txt")
E: activity-alias (line=122)
- A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100002
- A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.LightIcon" (Raw: "com.deniscerri.ytdl.LightIcon")
- A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=false
+ A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100000
+ A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.Default" (Raw: "com.deniscerri.ytdl.Default")
+ A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=true
A: http://schemas.android.com/apk/res/android:exported(0x01010010)=true
A: http://schemas.android.com/apk/res/android:launchMode(0x0101001d)=2
A: http://schemas.android.com/apk/res/android:configChanges(0x0101001f)=0x00000004
A: http://schemas.android.com/apk/res/android:targetActivity(0x01010202)="com.deniscerri.ytdl.MainActivity" (Raw: "com.deniscerri.ytdl.MainActivity")
A: http://schemas.android.com/apk/res/android:windowSoftInputMode(0x0101022b)=0x00000020
- A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100005
+ A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100003
E: meta-data (line=132) Dex diff: magic : 'dex\n035\0'
-checksum : 8f9795f3
-signature : 4d8c...e13a
+checksum : ba619438
+signature : 4589...800b
file_size : 10043656
header_size : 112
link_size : 0
@@ -409077,7 +409077,7 @@
name : 'VERSION_CODE'
type : 'I'
access : 0x0019 (PUBLIC STATIC FINAL)
- value : 107090200
+ value : 107090204
#4 : (in Lcom/deniscerri/ytdl/BuildConfig;) So do we go with building everything to have RB now – or do we wait until we can find out how to properly build a single ABI with RB later? |
Let's go with how it is for now |
That manifest diff is very odd. I'm thinking this is a bug in the toolchain then. So yeah, made sense to try a few things first, but let's just use what works for now :) |
So with all agreeing, I'll now implement it. Done: |
I've checked your app if its build is reproducible (see: Reproducible bulds, special client support and more in our repo), but while I was able to successfully generate the APK using
./gradlew assembleRelease
, the differences to the one provided at your latest release were huge. Was that APK really built from the commit the tag points to? If so, did I miss some build options? And if not, which commit was it?APK Diff:
We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.
Looking forward to your reply!
The text was updated successfully, but these errors were encountered: