Hi,

I skipped the report last month for lack of time, so the delta below is for two months.
I managed to do some patches and bug-reports. Mostly mundane stuff.

The most noteworthy thing is the fallout from https://github.com/rpm-software-management/rpm/pull/2047 that adds the number of build machine cores to many rpms, so it gets harder to bit-reproduce.

And the auto-generated report:

last month's status:
https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/GFJP5HGSAPHJ4S63D3PYQJ237EDCMBXQ/

Last months' reproducible builds project updates (including my work):
https://reproducible-builds.org/reports/2022-12/

I uploaded https://rb.zq1.de/compare.factory-20230109/ today

https://rb.zq1.de/spec/glossar.txt explains the meaning of below values:
total-packages: 14678 (+100)
build-tried: 14664 (+99)
build-failed: 33 (-12)
build-n-a: 225 (-2)
build-succeeded: 14405 (+111)
build-official-failed+na: 17 (+0)
build-compare-failed: 562 (-13)
build-compare-succeeded: 13843 (+124)
verify-failed: 401 (-18)
verified-semi-reproducible: 11672 (-5)
bit-by-bit-identical: 13241 (-249)
not-bit-by-bit-identical: 1165 (+365)
not-bit-by-bit-identicalcheck: 1164 (+360)

https://rb.zq1.de/compare.factory-20230109/graph.png
shows the change over time

https://rb.zq1.de/compare.factory-20230109/unreproduciblerings.txt
lists very unreproducible core packages (bootstrap+DVD)

Of the badly unreproducible packages,
5 were in ring0
78 were in ring1

That makes it 83/3316 => 2.50 %
which is below the overall average of
562/14405 => 3.90 %

1165/14405 => 8.09 % of packages are not perfectly reproducible


package notes:

==> ./ansible/.rb.notes <==
python3 toolchain pyc ASLR

==> ./apache-commons-dbcp/.rb.notes <==
javadoc toolchain

==> ./apache-commons-vfs2/.rb.notes <==
javadoc toolchain
/usr/share/javadoc/apache-commons-vfs2/deprecated-list.html differs (HTML document, ASCII text)

==> ./aqute-bnd/.rb.notes <==
javadoc toolchain html order issues
/usr/share/javadoc/aqute-bnd/biz.aQute.bndlib/index-all.html differs (HTML document, ASCII text, with very long lines)

==> ./asymptote/.rb.notes <==
=> https://github.com/vectorgraphics/asymptote/pull/351 date ; remaining pdf issue


==> ./b4/.rb.notes <==
python3 toolchain .pyc ASLR 
old: parallelism in tar.xz from _service recompress at buildtime

==> ./bin86/.rb.notes <==
date 2038-overflow


==> ./bmap-tools/.rb.notes <==
FTBFS-2024-06-12


==> ./borgbackup/.rb.notes <==
tar/gz?
= https://github.com/borgbackup/borg/issues/6996 FTBFS-2038

==> ./checkstyle/.rb.notes <==
date+time ; xmvn random uuid
+++ new//usr/share/java/checkstylecompilation.properties        2020-03-09 23:52:37.968994653 +0000

==> ./dwz/.rb.notes <==
=> 1036953 PGO+parallelism only
in ./dwz PGO training call even with taskset 1

==> ./dxvk/.rb.notes <==
?timestamp? in dlls


==> ./elixir/.rb.notes <==
= https://bugzilla.opensuse.org/show_bug.cgi?id=1205134 FTBFS-j1 stuck
osc build --vm-type=kvm --clean --noservice -j1 standard

==> ./endeavour/.rb.notes <==
= https://gitlab.gnome.org/World/Endeavour/-/issues/503 report FTBFS-j1
osc build --vm-type=kvm -j1 --clean --noservice standard

==> ./form/.rb.notes <==
FTBFS-j1
osc build --vm-type=kvm -j1 --noservice standard

==> ./himalaya/.rb.notes <==
random | rust
+++ new /usr/bin/himalaya (objdump)

==> ./icmake/.rb.notes <==
=> https://gitlab.com/fbb-git/icmake/-/merge_requests/5 sort find readdir order
from

==> ./indriya/.rb.notes <==
java date+time
+++ new//usr/share/java/indriya/META-INF/MANIFEST.MF    2020-03-09 19:59:46.201491946 +0000

==> ./java-17-openj9/.rb.notes <==
Java - many issues
/usr/lib64/jvm/java-17-openj9-17/lib/OMRTraceFormat.dat differs

==> ./java-18-openjdk/.rb.notes <==
Java - many issues
/usr/lib64/jvm/java-18-openjdk-18/lib/server/classes_nocoops.jsa differs at offset '5' (data)

==> ./ldap-sdk/.rb.notes <==
javadoc toolchain

==> ./libarchive/.rb.notes <==
= https://github.com/libarchive/libarchive/issues/1837 FTBFS-2038
=> https://github.com/libarchive/libarchive/pull/1836 FTBFS-2038

==> ./libnvme/.rb.notes <==
=> SR 1032570 date
+++ new//usr/share/man/man2/nvmf_update_config.2        2022-09-01 12:23:50.953451512 +0000

==> ./libspf2/.rb.notes <==
doxygen toolchain

==> ./llmnrd/.rb.notes <==
=> SR 1037221 minor filesys or toolchain


==> ./morfologik-stemming/.rb.notes <==
xmvn random uuid, date+time
+++ new//usr/share/java/morfologik-stemming/morfologik-fsa-builders.jar/META-INF/maven/org.carrot2/morfologik-fsa-builders/pom.properties	2022-12-10 14:42:44.247061412 +0000

==> ./muparserx/.rb.notes <==
ASLR
into /usr/share/doc/packages/muparserx-devel/html/inherit_graph_3.png

==> ./mybatis/.rb.notes <==
java/maven filesys strip-nd, date+time
/usr/share/java/mybatis.jar/META-INF/MANIFEST.MF differs (ASCII text, with CRLF line terminators)

==> ./nim/.rb.notes <==
-j1 failed


==> ./opennlp/.rb.notes <==
maven jar strip-nd , date+time
/usr/share/java/opennlp/opennlp-tools.jar/META-INF/maven/org.apache.opennlp/opennlp-tools/pom.properties differs (ASCII text)

==> ./OpenRGB/.rb.notes <==
=> https://gitlab.com/CalcProgrammer1/OpenRGB/-/merge_requests/1567 filesys readdir order


==> ./opensaml/.rb.notes <==
doxygen-1.9.5 toolchain readdir order

==> ./openscap/.rb.notes <==
= https://github.com/OpenSCAP/openscap/issues/1909 FTBFS
=> https://github.com/OpenSCAP/openscap/commit/8dec1bb5e9546e75ae6e7b7cf94cf00197ce3e5d

==> ./ouch/.rb.notes <==
rust libzstd_sys read_dir order => https://github.com/gyscos/zstd-rs/commit/6fe67d6a

==> ./pdsh/.rb.notes <==
minor rpm metadata
filterdiff printrpmtags RPMS*/*src.rpm|grep ^-

==> ./perl-HTTP-Cookies/.rb.notes <==
=> https://github.com/libwww-perl/HTTP-Cookies/pull/72 FTBFS-2038


==> ./python-aiosmtplib/.rb.notes <==
FTBFS-2038 SSL
found range good=2145916633 bad=2145918558 (2038-01-01T00:29:18)

==> ./python-arrow/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-asyncpg/.rb.notes <==
found range good=1956864895 bad=1956866820 (2032-01-04T22:07:00)
osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-05-26T18:00:00" standard # Fails

==> ./python-azure-storage-file-share/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-bandit/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-devpi-server/.rb.notes <==
= https://github.com/devpi/devpi/issues/948 FTBFS -j1 failed
osc build --clean --vm-type=kvm -j1 --noservice standard

==> ./python-enaml/.rb.notes <==
=> https://github.com/nucleic/enaml/pull/516 FTBFS-2038
osc build --vm-type=kvm --noservice --clean  --build-opt=--vm-custom-opt="-rtc base=2038-02-01T18:00:00" standard

==> ./python-Faker/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-fsspec/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-gspread/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-gwpy/.rb.notes <==
= failing from astropy leap second expiry warning https://github.com/astropy/astropy/issues/10228
--build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed

==> ./python-holoviews/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-maturin/.rb.notes <==
#= https://bugzilla.opensuse.org/show_bug.cgi?id=1206342 report rust random
8 random bytes from vendor/ahash-0.3.8/src/random_state.rs into /usr/bin/maturin-3.8 differs in ELF sections

==> ./python-mautrix/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-mitmproxy/.rb.notes <==
=> https://github.com/mitmproxy/mitmproxy/pull/5852 FTBFS-2037

==> ./python-notebook/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-onionshare/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-pip/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-pre-commit/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-psutil/.rb.notes <==
ASLR python ; stuck -j4?


==> ./python-pycryptodomex/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-pycsw/.rb.notes <==
--build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed
found range good=1673068535 bad=1673070459 (2023-01-07T05:47:39)

==> ./python-PyInstaller/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-pypandoc/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-pyprimes/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-pytzdata/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-tablib/.rb.notes <==
python3 .pyc ASLR

==> ./python-tasklib/.rb.notes <==
--build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed


==> ./python-Telethon/.rb.notes <==
python3 pyc toolchain ASLR


==> ./python-toolz/.rb.notes <==
python3 pyc toolchain ASLR

==> ./python-trio/.rb.notes <==
python3 pyc toolchain ASLR


==> ./qca/.rb.notes <==
FTBFS-2031
found range good=1672719473 bad=1672721397 (2023-01-03T04:49:57)

==> ./qemu/.rb.notes <==
/usr/share/qemu/slof.bin differs at offset '108' (data)


==> ./qt6-positioning/.rb.notes <==
filesystem+ASLR+parallelism
/usr/lib64/qt6/examples/positioning/weatherinfo/weatherinfo differs in assembler output

==> ./relaxngcc/.rb.notes <==
minor strip-nondeterminism
-/usr/share/javadoc/relaxngcc/relaxngcc/util/ConcatIterator.html 

==> ./rnp/.rb.notes <==
= https://github.com/rnpgp/rnp/issues/1961 FTBFS-2024-01-19 from src/lib/sec_profile.cpp SHA-1 depecation
#found range good=1705629262 bad=1705631160 (2024-01-19T02:26:00)

==> ./rpm-config-SUSE/.rb.notes <==
date+time ; acceptable variation?
/usr/lib/rpm/macros.d/macros.opensuse differs (ASCII text)

==> ./rss2email/.rb.notes <==
python3 toolchain .pyc ASLR

==> ./sbsigntools/.rb.notes <==
filesys readdir ordering from automake
in /usr/bin/sbvarsign

==> ./sccache/.rb.notes <==
=> https://github.com/gyscos/zstd-rs/commit/6fe67d6a toolchain filesys readdir order ; needs release


==> ./shibboleth-sp/.rb.notes <==
doxygen-1.9.5 toolchain filesys readdir order

==> ./skaffold/.rb.notes <==
parallelism issue solved 20221108 with go 1.19.2


==> ./spotifyd/.rb.notes <==
random


==> ./spyder/.rb.notes <==
python3 toolchain pyc ASLR


==> ./syft/.rb.notes <==
go1.18 parallelism build-id
still problematic with go1.19.4

==> ./telegraf/.rb.notes <==
go parallelism .note.go.buildid:


==> ./teleport/.rb.notes <==
go parallelism .note.go.buildid:


==> ./texlab/.rb.notes <==
ASLR rust


==> ./thonny/.rb.notes <==
python3 toolchain .pyc ASLR

==> ./trivy/.rb.notes <==
parallelism  go1.18 - solved with go1.19.3?

==> ./uimaj/.rb.notes <==
xmvn random uuid ; date+time
+++ new//usr/lib64/java/uimaj/uimaj-core.jar/META-INF/MANIFEST.MF	2022-12-10 16:52:06.762983827 +0000

==> ./unit-api/.rb.notes <==
javadoc jar html filesys, strip-nd, date+time
+++ new//usr/share/java/unit-api.jar/META-INF/maven/javax.measure/unit-api/pom.properties	2022-12-10 17:06:16.577151499 +0000

==> ./uom-lib/.rb.notes <==
xmvn uuid, date+time
/usr/share/java/uom-lib/uom-lib-common.jar/META-INF/maven/tech.uom.lib/uom-lib-common/pom.properties differs (ASCII text)

==> ./vectorscan/.rb.notes <==
=> SR 1032506 CPU -march=
/usr/bin/unit-hyperscan differs in assembler output

==> ./waf/.rb.notes <==
time-based .pyc

==> ./warzone2100/.rb.notes <==
= https://github.com/Warzone2100/warzone2100/issues/2991 report parallelism-dependent output
- Compression succeeded to file "/home/abuild/rpmbuild/BUILD/warzone2100/build/data/base/texpages/page-102-bottom-rockies.ktx2" size 1375248 bytes in 144.393 secs

==> ./weblate/.rb.notes <==
= https://github.com/WeblateOrg/weblate/issues/8556 FTBFS-2038 toolchain mercurial
ERROR: test_upstream_changes_rename (weblate.vcs.tests.test_vcs.VCSHgTest)

==> ./xemacs-packages/.rb.notes <==
FTBFS-2038 stuck
osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-12-10T00:59:10" standard

==> ./xemacs/.rb.notes <==
FTBFS-2038 stuck
osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-12-10T00:59:10" standard

==> ./xmltooling/.rb.notes <==
doxygen toolchain filesys readdir order