...along with a bunch of incompatible API/ABI changes.
I had to make a few tweaks in the build process as well, and these
couldn't be made (and tested!) with separate commits. The test suite was
failing on OS X 13, so I decided to upgrade, because "why not".
First, let's try using GCC 12 on that platform. That's needed because
the GCC13 builds started failing like this:
/opt/homebrew/bin/gcc-13 -DLIBYANG_BUILD -I/Users/runner/work/oopt-gnpy-libyang/build-libyang/src -I/Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang/src -I/Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang/src/plugins_exts -I/Users/runner/work/oopt-gnpy-libyang/build-libyang/compat -I/opt/homebrew/include -DNDEBUG -O2 -Wall -Wextra -Wpedantic -std=c11 -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -mmacosx-version-min=13.0 -fPIC -fvisibility=hidden -MD -MT CMakeFiles/yangobj.dir/src/dict.c.o -MF CMakeFiles/yangobj.dir/src/dict.c.o.d -o CMakeFiles/yangobj.dir/src/dict.c.o -c /Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang/src/dict.c
In file included from /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/include/pthread.h:57,
from /Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang/src/dict.c:19:
/Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/time.h:198:67: error: macro "__API_AVAILABLE3" passed 4 arguments, but takes just 3
198 | __API_AVAILABLE(macosx(10.15), ios(13.0), tvos(13.0), watchos(6.0))
| ^
This gets fixed by using GCC 12, and with that, we can *build* the
`libyang` project. What we cannot do is *test* it, because the following
stuff started failing, apparently due to a change of `expect` version:
58/61 Test #61: yangre ............................***Failed 0.16 sec
Tests running in interp: /usr/bin/tclsh
Tests located in: /Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang/tests/yangre
Tests running in: /Users/runner/work/oopt-gnpy-libyang/build-libyang/tests/yangre
Temporary files stored in /Users/runner/work/oopt-gnpy-libyang/build-libyang/tests/yangre
Test files run in separate interpreters
Running tests that match: *
Skipping test files that match: l.*.test
Only running test files that match: *.test
Tests began at Wed Jun 19 09:48:46 UTC 2024
arg.test
==== arg_empty Missing arguments FAILED
==== Contents of test case:
ly_cmd_err "" "missing <string> parameter to process"
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: invalid command name "try"
while executing
"try {
set results [exec -- $TUT {*}$cmd]
set status 0
} trap CHILDSTATUS {results options} {
# return code is not 0
..."
(procedure "ly_exec" line 3)
invoked from within
"ly_exec $cmd"
(procedure "ly_cmd_err" line 3)
invoked from within
"ly_cmd_err "" "missing <string> parameter to process""
("uplevel" body line 2)
invoked from within
"uplevel 1 $script"
---- errorCode: NONE
==== arg_empty FAILED
Indeed, there's a small difference between these two image/sw versions:
- https://github.com/actions/runner-images/blob/macos-12/20240602.1/images/macos/macos-12-Readme.md
This one succeeds, and it says it has got a "Tcl/Tk 8.6.14" installed
- https://github.com/actions/runner-images/blob/macos-13/20240616.1/images/macos/macos-13-Readme.md
This one fails, and there's no mention of Tcl
So, fix this by always installing `tcl-tk` from Homebrew. As a bonus, we
can now drop the manual workaround for libyang tests "without expect".
Yay, one fewer FIXME.
At this point we have `libyang` the C library built, installed and
tested -- yay. It's time to move to the C++ code, which breaks with a
wonderful internal error in the linker:
FAILED: libyang-cpp.dylib
: && /usr/local/bin/g++-12 -Wall -Wextra -pedantic -Woverloaded-virtual -Wimplicit-fallthrough -Wsuggest-override -O3 -DNDEBUG -arch x86_64 -isysroot /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -mmacosx-version-min=13.0 -dynamiclib -Wl,-headerpad_max_install_names -o libyang-cpp.dylib -install_name @rpath/libyang-cpp.dylib CMakeFiles/yang-cpp.dir/src/ChildInstantiables.cpp.o CMakeFiles/yang-cpp.dir/src/Context.cpp.o CMakeFiles/yang-cpp.dir/src/DataNode.cpp.o CMakeFiles/yang-cpp.dir/src/Enum.cpp.o CMakeFiles/yang-cpp.dir/src/Collection.cpp.o CMakeFiles/yang-cpp.dir/src/Module.cpp.o CMakeFiles/yang-cpp.dir/src/SchemaNode.cpp.o CMakeFiles/yang-cpp.dir/src/Set.cpp.o CMakeFiles/yang-cpp.dir/src/Type.cpp.o CMakeFiles/yang-cpp.dir/src/Utils.cpp.o CMakeFiles/yang-cpp.dir/src/utils/exception.cpp.o CMakeFiles/yang-cpp.dir/src/utils/ref_count.cpp.o CMakeFiles/yang-cpp.dir/src/utils/newPath.cpp.o -Wl,-rpath,/Users/runner/work/oopt-gnpy-libyang/target/lib /Users/runner/work/oopt-gnpy-libyang/target/lib/libyang.dylib && :
-macosx_version_min has been renamed to -macos_version_min
0 0x10dd0cf43 __assert_rtn + 64
1 0x10dc0ef43 ld::AtomPlacement::findAtom(unsigned char, unsigned long long, ld::AtomPlacement::AtomLoc const*&, long long&) const + 1411
2 0x10dc2b431 ld::InputFiles::SliceParser::parseObjectFile(mach_o::Header const*) const + 19745
3 0x10dc3bb71 ld::InputFiles::parseAllFiles(void (ld::AtomFile const*) block_pointer)::$_7::operator()(unsigned long, ld::FileInfo const&) const + 657
4 0x7ff806e9f066 _dispatch_client_callout2 + 8
5 0x7ff806eb218f _dispatch_apply_invoke_and_wait + 213
6 0x7ff806eb1692 _dispatch_apply_with_attr_f + 1207
7 0x7ff806eb1847 dispatch_apply + 45
8 0x10dcd4972 ld::AtomFileConsolidator::parseFiles(bool) + 370
9 0x10dc5bd67 main + 12263
ld: Assertion failed: (resultIndex < sectData.atoms.size()), function findAtom, file Relocations.cpp, line 1336.
collect2: error: ld returned 1 exit status
Now, this looks a bug in the particular version of XCode that's used in
this particular release of the Mac OS X GitHub CI runner, and the fix is
apparently just to use a different version. That version however
apparently cannot be the 15.1, because C++ exceptions are apparently
FUBAR in there:
1/4 Test #1: test_context .....................Subprocess aborted***Exception: 0.02 sec
libyang[0]: Unexpected end-of-input. (path: Line number 1.)
terminate called after throwing an instance of 'libyang::ErrorWithCode'
what(): Can't parse module: LY_EVALID
[doctest] doctest version is "2.4.11"
[doctest] run with "--help" for options
===============================================================================
/Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang-cpp/tests/context.cpp:66:
TEST CASE: context
parseModule
invalid
/Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang-cpp/tests/context.cpp:66: FATAL ERROR: test case CRASHED: SIGABRT - Abort (abnormal termination) signal
===============================================================================
/Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/libyang-cpp/tests/context.cpp:66:
TEST CASE: context
DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
parseModule
invalid
===============================================================================
[doctest] test cases: 1 | 0 passed | 1 failed | 1 skipped
[doctest] assertions: 4 | 4 passed | 0 failed |
[doctest] Status: FAILURE!
2/4 Test #4: test_unsafe ......................***Exception: SegFault 0.02 sec
libunwind: _Unwind_GetDataRelBase - _Unwind_GetDataRelBase() not implemented
At that point everything builds, but macOS 12 wheels fail to install.
That's because "something" has changed in the CI images, and the
provided Python interpreter does not use the 12_04 tag, but it instead
stops at version 12_0, like this:
$ pip install -vvv --only-binary :all: --no-index --find-links=${GITHUB_WORKSPACE//\\//}/wheelhouse oopt-gnpy-libyang
...
Skipping link: none of the wheel's tags (cp311-cp311-macosx_12_4_x86_64) are compatible (run pip debug --verbose to show compatible tags): file:///Users/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/wheelhouse/oopt_gnpy_libyang-0.0.15.dev1%2Bga16eba9-cp311-cp311-macosx_12_4_x86_64.whl
...
$ pip debug --verbose
...
Compatible tags: 2264
cp311-cp311-macosx_12_0_x86_64
cp311-cp311-macosx_12_0_intel
cp311-cp311-macosx_12_0_fat64
cp311-cp311-macosx_12_0_fat32
cp311-cp311-macosx_12_0_universal2
cp311-cp311-macosx_12_0_universal
...
Since there's been no reason given on why we might need version 12.4
instead of just 12.0 in the original commit which added support for Mac
OS buidls in the first place, let's just call this 12.0 and hope that
the ABI overlords which control the C++ STL are on our side.
Bug: https://github.com/actions/runner-images/issues/9997
Bug: https://forums.developer.apple.com/forums/thread/737707
Bug: https://github.com/actions/runner-images/issues/9273
Bug: https://github.com/Homebrew/homebrew-core/issues/145991
Bug: https://stackoverflow.com/questions/78017671/runtime-issues-with-stdoptional-in-gcc-on-m2-mac
Warning, Context.get_module's API changed, the revision no longer
default to `None` because passing `None` explicitly asks for a module
with no associated revision. Use `Context.get_module_latest` when the
previous behavior is needed. This is a deliberate change from the C++
library; only in C++ a compiler catches this :).
See-also: https://gerrit.cesnet.cz/c/CzechLight/libyang-cpp/+/6732
See-also: https://gerrit.cesnet.cz/c/CzechLight/libyang-cpp/+/6733
...so that we can build a native binary wheel for this platform.
Technically speaking, cross-builds should have been possible "for
years", but I was not able to get one running successfully (see that
removed comment). Let's go native.
This CI runner is currently a paid one, so forks of this repo might not
pass all actions.
I think this is a reasonable thing which should limit the number of
wasted CPU cycles. If someone can upgrade to a new enough Python, let's
hope they can update to a new enough OS image to run it.
There are some failures with `gnpy` on Windows:
================================== FAILURES ===================================
____________ test_lint_yang[ietf-network-topology@2018-02-26.yang] ____________
yang_model = WindowsPath('D:/a/oopt-gnpy/oopt-gnpy/gnpy/yang/ext/ietf-network-topology@2018-02-26.yang')
@pytest.mark.parametrize("yang_model", [x for x in external_path().glob('*.yang')] + [x for x in model_path().glob('*.yang')], ids=_get_basename)
def test_lint_yang(yang_model):
'''Run a linter on each YANG model'''
c = ly.Context(str(external_path()) + os.pathsep + str(model_path()),
ly.ContextOptions.NoYangLibrary | ly.ContextOptions.DisableSearchCwd)
assert c.parse_module(yang_model, ly.SchemaFormat.YANG) is not None
> assert c.errors() == []
E assert <[TypeError("unhashable type: 'instancemethod'") raised in repr()] list object at 0x1880e5958c0> == []
E (pytest_assertion plugin: representation of details failed: C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\pprint.py:178: TypeError: unhashable type: 'instancemethod'.
E Probably an object has a faulty __repr__.)
tests\test_yang_lint.py:30: AssertionError
---------------------------- Captured stderr call -----------------------------
libyang[1]: File name "D:\a\oopt-gnpy\oopt-gnpy\gnpy\yang\ext\ietf-network-topology@2018-02-26.yang" does not match module name "ietf-network-topology".
________________ test_lint_yang[ietf-network@2018-02-26.yang] _________________
That complaint about __repr__ is an impedance mismatch between pybind11
and Python (pybind/pybind11#2722), but in this case that whole thing is
triggered through a bug in libyang which assumes that the directory
separator is always `/`.
Bug: https://github.com/CESNET/libyang/pull/2093
I'm deliberately not checking whether the headers are self-contained
because it seems that MSVC has different rules about
std::optional/std::variant intrinsic traits compared to GCC and clang.
Here's how to reproduce via GitHub actions:
cmake --build . --target=all_verify_interface_header_sets -j${{ steps.cpu-cores.outputs.count }} --config ${{ matrix.build_type }}
On Windows and OS X, this simply adds a bunch of extra builds. On Linux,
I'm also bumping the manylinux tag because Python 3.11 is newer than the
distro in question. This means that there will be no builds for "new
Python on old distros" because one has to draw a line somewhere.
Effectively, wheels for Linux Python 3.11 will therefore require glibc
2.35.
I don't have the details about why exactly that version of pybind11 on
Ubuntu 20.04 broke down, but here's how it ended up on Ubuntu 22.04 when
trying to use Python 3.11:
In file included from /usr/include/pybind11/cast.h:16,
from /usr/include/pybind11/attr.h:13,
from /usr/include/pybind11/pybind11.h:13,
from /usr/include/pybind11/functional.h:12,
from /home/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/oopt-gnpy-libyang.cpp:1:
/usr/include/pybind11/detail/type_caster_base.h: In function ‘std::string pybind11::detail::error_string()’:
/usr/include/pybind11/detail/type_caster_base.h:482:26: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
482 | frame = frame->f_back;
| ^~
In file included from /opt/hostedtoolcache/Python/3.11.1/x64/include/python3.11/Python.h:42,
from /usr/include/pybind11/detail/common.h:215,
from /usr/include/pybind11/pytypes.h:12,
from /usr/include/pybind11/cast.h:13,
from /usr/include/pybind11/attr.h:13,
from /usr/include/pybind11/pybind11.h:13,
from /usr/include/pybind11/functional.h:12,
from /home/runner/work/oopt-gnpy-libyang/oopt-gnpy-libyang/oopt-gnpy-libyang.cpp:1:
/opt/hostedtoolcache/Python/3.11.1/x64/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
On Windows, it failed because (apparently) vcpkg patches Python
detection as per https://github.com/microsoft/vcpkg/issues/23796. The
build would fail like this (vcpkg Pytohn is 3.10, the system-selected
one is 3.11):
CMake Error at C:/t/build-env-1a5f954n/Lib/site-packages/cmake/data/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python (missing: Development.Module) (found suitable version
"3.11.1", minimum required is "3.8")
Call Stack (most recent call first):
C:/t/build-env-1a5f954n/Lib/site-packages/cmake/data/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
C:/t/build-env-1a5f954n/Lib/site-packages/cmake/data/share/cmake-3.25/Modules/FindPython/Support.cmake:3245 (find_package_handle_standard_args)
C:/t/build-env-1a5f954n/Lib/site-packages/cmake/data/share/cmake-3.25/Modules/FindPython.cmake:519 (include)
So let's just use the PIP magic and hope for the best.
