Build is green
See https://jenkins.softwareheritage.org/job/DLDG/job/tox/95/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DLDG/job/tox/96/ for more details.

https://forge.softwareheritage.org/P429
this is the error message from one of the tests. Results in all py3 tests failing.

I'm surprised since [1] is currently (without the diff's content i mean) green.
It has been for a while ;)

[1] https://jenkins.softwareheritage.org/job/DLDG/job/tests/

Ok, so like i said in irc, pyblake2 is a transitive dependency of the loader-git through the swh-model (in charge of computing hashes).
So we should not have to declare that here.

So the problem you are facing must lie elsewhere.

Have you tried to recreate the tox environment (prior to this commit)?

tox --recreate

I was clearing existing tox setup with rm -rf .tox, when I changed requirements. I will try tests of other modules as well and report if I find similar errors.

.tox/py3/lib/python3.5/site-packages/swh/model/hashutil.py:211: in _new_hashlib_hash
    return _new_blake2_hash(algo)

def _new_blake2_hash(algo):
        """Return a function that initializes a blake2 hash.
    
        """
    
        < -- removed for clarity -->
    
        if lalgo in hashlib.algorithms_available:
            # Handle the case where OpenSSL ships the given algorithm
            # (e.g. Python 3.5 on Debian 9 stretch)
            _blake2_hash_cache[algo] = lambda: hashlib.new(lalgo)
        else:
            # Try using the built-in implementation for Python 3.6+
            if blake_family in hashlib.algorithms_available:
                blake2 = getattr(hashlib, blake_family)
            else:
>               import pyblake2
E               ImportError: No module named 'pyblake2'

As you can see I am using python3.5 which is the default version installed as per instructions in developer setup. But the code assumes Python 3.6+ when it does not find lalgo. I am not sure why it is working everywhere but not on my setup.

.tox/py3/lib/python3.5/site-packages/swh/model/hashutil.py:211: in _new_hashlib_hash
    return _new_blake2_hash(algo)

def _new_blake2_hash(algo):
        """Return a function that initializes a blake2 hash.
    
        """
    
        < -- removed for clarity -->
    
        if lalgo in hashlib.algorithms_available:
            # Handle the case where OpenSSL ships the given algorithm
            # (e.g. Python 3.5 on Debian 9 stretch)
            _blake2_hash_cache[algo] = lambda: hashlib.new(lalgo)
        else:
            # Try using the built-in implementation for Python 3.6+
            if blake_family in hashlib.algorithms_available:
                blake2 = getattr(hashlib, blake_family)
            else:
>               import pyblake2
E               ImportError: No module named 'pyblake2'

As you can see I am using python3.5 which is the default version installed as per instructions in developer setup. But the code assumes Python 3.6+ when it does not find lalgo. I am not sure why it is working everywhere but not on my setup.

It seems like there is something fishy indeed.

What distribution are you using (for understanding and reproductibility concerns)?

In any case, the problem lies in swh-model repository though.
We can finish discussing it here first though, we'll reference back if we need to.

I am on a standard Ubuntu 16.04 Xenial distribution. Is there any specific config files you want to see.

I am on a standard Ubuntu 16.04 Xenial distribution. Is there any specific config files you want to see.

Thanks.

In this context, come to think of it (more), that should be irrelevant... Since we are using tox (thus pip and pypi underneath)...

In D1574#35522, @twitu wrote:

I am not sure why it is working everywhere but not on my setup.

I am guessing this is because we use Debian 9 everywhere, whose openssl supports blake2s256.

I don't want to look for that information later so here is the excerpt from @olasd on irc (which sounds totally legit, though i did not check further than that):

14:00 <+olasd> so, twitu[m]'s issue is that : swh.loader.git depends on swh.model; swh.model has been uploaded as a platform-independent wheel to PyPI; swh.model has setup.py magic to detect whether pyblake2 should be installed; we built the wheel on a platform that doesn't need pyblake2 to be installed; installing a wheel doesn't run setup.py
14:01 <+olasd> (but uses requires information from the build platform for dependency resolution)

I don't know yet how to fix properly the issue (again the fix is due in swh-model anyway).

A middle-ground would be to change swh-model to install pyblake2 on all systems with a Python < 3.6, by using this in the dependency declaration: extras_require={':python_version<"3.6"':: ['pyblake2']}

That sounds completely reasonable to me.
Thanks for the idea.

	blake2_requirements = []
	
	pyblake2_hash_sets = [
	    # Built-in implementation in Python 3.6+
	    {'blake2s', 'blake2b'},
	    # Potentially shipped by OpenSSL 1.1 (e.g. Python 3.5 in Debian stretch
	    # has these)
	    {'blake2s256', 'blake2b512'},
	]
	
	for pyblake2_hashes in pyblake2_hash_sets:
	    if not pyblake2_hashes - set(hashlib.algorithms_available):
	        # The required blake2 hashes have been found
	        break
	else:
	    # None of the possible sets of blake2 hashes are available.
	    # use pyblake2 instead
	    blake2_requirements.append('pyblake2')

This snippet from setup.py in swh-model adds pyblake2 to blake2_requirements in my local setup. Shouldn't be already installed then?

The setup.py is not run when you install a wheel. It's pre-computed by the computer that compiles the wheel, not yours.

Build is green
See https://jenkins.softwareheritage.org/job/DMOD/job/tox/86/ for more details.

I built and tested the wheel locally, it did install pyblake2 as per the requirements.

@twitu The issue is when the wheel is built on a computer where openssl supports blake2 and installed on a computer where openssl does not support it.

Build is green
See https://jenkins.softwareheritage.org/job/DMOD/job/tox/88/ for more details.

Foolishly committed to master earlier, created new branch now

Build is green
See https://jenkins.softwareheritage.org/job/DMOD/job/tox/89/ for more details.

Thanks for the fix!

Foolishly committed to master earlier, created new branch now

That's not foolish.
And that's not an issue either.
In the end, it will land in the repository's default branch (mostly master).

My workflow is mostly to work on the master branch and diff against it.
Then i rebase my local code when the repository on that branch progresses.
And then i update the diff (prior to ci's build ok and push).

When i work on a branch, i open a diff against that branch.
In the end, when the diff is accepted.
I update the master branch with the latest modification from the team.
I merge (as a fast-forward) my local branch to master.
Then i still update the diff, wait for the ci to be ok, then i push.

That has the benefit to close automatically the diff in question.

Cheers,

Build is green
See https://jenkins.softwareheritage.org/job/DMOD/job/tox/90/ for more details.