Page MenuHomeSoftware Heritage

metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
Closed, MigratedEdits Locked

Description

It happens on staging after the activation of the indexation of the metadata generated by the indexers :

Dec 10 11:12:25 search0 swh[31455]: INFO:elasticsearch:POST http://search-esnode0.internal.staging.swh.network:9200/origin/_bulk [status:200 request:0.096s]
Dec 10 11:12:25 search0 swh[31455]: Traceback (most recent call last):
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/bin/swh", line 11, in <module>
Dec 10 11:12:25 search0 swh[31455]:     load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
Dec 10 11:12:25 search0 swh[31455]:     return swh(auto_envvar_prefix="SWH")
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
Dec 10 11:12:25 search0 swh[31455]:     return self.main(*args, **kwargs)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
Dec 10 11:12:25 search0 swh[31455]:     rv = self.invoke(ctx)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Dec 10 11:12:25 search0 swh[31455]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Dec 10 11:12:25 search0 swh[31455]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Dec 10 11:12:25 search0 swh[31455]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
Dec 10 11:12:25 search0 swh[31455]:     return ctx.invoke(self.callback, **ctx.params)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
Dec 10 11:12:25 search0 swh[31455]:     return callback(*args, **kwargs)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
Dec 10 11:12:25 search0 swh[31455]:     return f(get_current_context(), *args, **kwargs)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/search/cli.py", line 102, in journal_client_objects
Dec 10 11:12:25 search0 swh[31455]:     nb_messages = client.process(worker_fn)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 265, in process
Dec 10 11:12:25 search0 swh[31455]:     batch_processed, at_eof = self.handle_messages(messages, worker_fn)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 292, in handle_messages
Dec 10 11:12:25 search0 swh[31455]:     worker_fn(dict(objects))
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/search/journal_client.py", line 31, in process_journal_objects
Dec 10 11:12:25 search0 swh[31455]:     process_origin_intrinsic_metadata(messages["origin_intrinsic_metadata"], search)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/search/journal_client.py", line 77, in process_origin_intrinsic_metadata
Dec 10 11:12:25 search0 swh[31455]:     search.origin_update(origin_metadata)
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/swh/search/elasticsearch.py", line 106, in origin_update
Dec 10 11:12:25 search0 swh[31455]:     bulk(self._backend, actions, index="origin")
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/elasticsearch/helpers/actions.py", line 300, in bulk
Dec 10 11:12:25 search0 swh[31455]:     for ok, item in streaming_bulk(client, actions, *args, **kwargs):
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/elasticsearch/helpers/actions.py", line 230, in streaming_bulk
Dec 10 11:12:25 search0 swh[31455]:     **kwargs
Dec 10 11:12:25 search0 swh[31455]:   File "/usr/lib/python3/dist-packages/elasticsearch/helpers/actions.py", line 158, in _process_bulk_chunk
Dec 10 11:12:25 search0 swh[31455]:     raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
Dec 10 11:12:25 search0 swh[31455]: elasticsearch.helpers.errors.BulkIndexError: ('1 document(s) failed to index.', [{'update': {'_index': 'origin', '_type': '_doc', '_id': '81eff50ed4f9d136e93da51c503054df62c0a0f0', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': 'object mapping for [intrinsic_metadata.author.affiliation] tried to parse field [affiliation] as object, but found a concrete value'}, 'data': {'doc': {'url': 'https://pypi.org/project/brightway2/', 'intrinsic_metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0', 'type': ('Code', 'SoftwareSourceCode'), 'author': ({'id': '0000-0002-7898-9862', 'type': 'Person', 'affiliation': 'Paul Scherrer Institut', 'email': 'cmutel@gmail.com', 'name': 'Chris Mutel'}, {'type': 'Person', 'email': 'cmutel@gmail.com', 'name': 'Chris Mutel'}), 'codeRepository': 'https://bitbucket.org/cmutel/brightway2', 'dateCreated': '2017-04-05', 'dateModified': '2017-04-05', 'datePublished': '2017-04-05', 'description': 'Framework for Life Cycle Assessment', 'identifier': './', 'keywords': 'LCA, Python', 'license': ('BSD 3-clause', 'Copyright (c) 2016, Chris Mutel and ETH Zurich.'), 'name': 'brightway2', 'url': 'https://bitbucket.org/cmutel/brightway2', 'version': ('2.0.2', '2.3')}, 'sha1': '81eff50ed4f9d136e93da51c503054df62c0a0f0'}, 'doc_as_upsert': True}}}])
Dec 10 11:12:25 search0 systemd[1]: swh-search-journal-client@indexed.service: Main process exited, code=exited, status=1/FAILURE

It seems there is no explicit mapping declared and the one created by ES is not generic enough :

"author": {
          "properties": {
            "affiliation": {
              "properties": {
                "name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },

complete mapping of the index:


generated with :

root@search-esnode0:/srv/elasticsearch# curl http://${ES_SERVER}/origin/_mapping\?pretty | jq '.origin.mappings' > /tmp/new_mapping.js

Related Objects

Event Timeline

vsellier triaged this task as Normal priority.Dec 10 2020, 12:31 PM
vsellier created this task.
vlorentz renamed this task from metadata indexation : error on affiliations to metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types.Dec 10 2020, 12:45 PM
vlorentz claimed this task.

We tried to change the mapping type of the field intrinsic_metadata from nested to flattened as you have suggested, we have now a new error related to the huge size of a description.
ES can be configured to accept bigger fields but I'm not sure it's relevant regarding the description field content.

root@search-esnode0:~# diff -U3 /tmp/mapping-origin.json /tmp/mapping.json 
--- /tmp/mapping-origin.json	2020-12-10 13:48:44.632000000 +0000
+++ /tmp/mapping.json	2020-12-10 13:54:33.948000000 +0000
@@ -4,12 +4,7 @@
       "type": "boolean"
     },
     "intrinsic_metadata": {
-      "type": "nested",
-      "properties": {
-        "@context": {
-          "type": "keyword"
-        }
-      }
+      "type": "flattened"
     },
     "sha1": {
       "type": "keyword"
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Mapping definition for [intrinsic_metadata] has unsupported parameters:  [properties : {@context={type=keyword}}]"}],"type":"mapper_parsing_exception","reason":"Mapping definition for [intrinsic_metadata] has unsupported parameters:  [properties : {@context={type=keyword...
ec 10 13:59:42 search0 swh[3329]: INFO:elasticsearch:POST http://search-esnode0.internal.staging.swh.network:9200/origin/_bulk [status:200 request:0.232s]
Dec 10 13:59:42 search0 swh[3329]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_ALL_BROKERS_DOWN,val=-187,str="1/1 brokers are down"}
Dec 10 13:59:42 search0 swh[3329]: Traceback (most recent call last):
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/bin/swh", line 11, in <module>
Dec 10 13:59:42 search0 swh[3329]:     load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
Dec 10 13:59:42 search0 swh[3329]:     return swh(auto_envvar_prefix="SWH")
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
Dec 10 13:59:42 search0 swh[3329]:     return self.main(*args, **kwargs)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
Dec 10 13:59:42 search0 swh[3329]:     rv = self.invoke(ctx)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Dec 10 13:59:42 search0 swh[3329]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Dec 10 13:59:42 search0 swh[3329]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Dec 10 13:59:42 search0 swh[3329]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
Dec 10 13:59:42 search0 swh[3329]:     return ctx.invoke(self.callback, **ctx.params)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
Dec 10 13:59:42 search0 swh[3329]:     return callback(*args, **kwargs)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
Dec 10 13:59:42 search0 swh[3329]:     return f(get_current_context(), *args, **kwargs)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/search/cli.py", line 102, in journal_client_objects
Dec 10 13:59:42 search0 swh[3329]:     nb_messages = client.process(worker_fn)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 265, in process
Dec 10 13:59:42 search0 swh[3329]:     batch_processed, at_eof = self.handle_messages(messages, worker_fn)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 292, in handle_messages
Dec 10 13:59:42 search0 swh[3329]:     worker_fn(dict(objects))
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/search/journal_client.py", line 31, in process_journal_objects
Dec 10 13:59:42 search0 swh[3329]:     process_origin_intrinsic_metadata(messages["origin_intrinsic_metadata"], search)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/search/journal_client.py", line 77, in process_origin_intrinsic_metadata
Dec 10 13:59:42 search0 swh[3329]:     search.origin_update(origin_metadata)
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/swh/search/elasticsearch.py", line 106, in origin_update
Dec 10 13:59:42 search0 swh[3329]:     bulk(self._backend, actions, index="origin")
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/elasticsearch/helpers/actions.py", line 300, in bulk
Dec 10 13:59:42 search0 swh[3329]:     for ok, item in streaming_bulk(client, actions, *args, **kwargs):
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/elasticsearch/helpers/actions.py", line 230, in streaming_bulk
Dec 10 13:59:42 search0 swh[3329]:     **kwargs
Dec 10 13:59:42 search0 swh[3329]:   File "/usr/lib/python3/dist-packages/elasticsearch/helpers/actions.py", line 158, in _process_bulk_chunk
Dec 10 13:59:42 search0 swh[3329]:     raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
Dec 10 13:59:42 search0 swh[3329]: elasticsearch.helpers.errors.BulkIndexError: ('1 document(s) failed to index.', [{'update': {'_index': 'origin', '_type': '_doc', '_id': '2a6712f5ddc72c54fb9f6aabab9cd5feb7e7eb13', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Document contains at least one immense term in field="intrinsic_metadata" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: \'[112, 121, 50, 97, 112, 112, 32, 105, 115, 32, 97, 32, 80, 121, 116, 104, 111, 110, 32, 115, 101, 116, 117, 112, 116, 111, 111, 108, 115, 32]...\', original message: bytes can be at most 32766 in length; got 52186', 'caused_by': {'type': 'max_bytes_length_exceeded_exception', 'reason': 'bytes can be at most 32766 in length; got 52186'}}, 'data': {'doc': {'url': 'https://pypi.org/project/py2app/', 'intrinsic_metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0', 'type': 'SoftwareSourceCode', 'author': ({'type': 'Person', 'email': 'ronaldoussoren@mac.com', 'name': 'Ronald Oussoren'},), 'description': ('Create standalone Mac OS X applications with Python', 'py2app is a Python setuptools command which will allow\nyou to make standalone Mac OS X application bundles\nand plugins from Python scripts.\n\npy2app is similar in purpose and design to py2exe for\nWindows.\n\nNOTE: py2app must be used on OSX to build applications,\nit cannot create Mac applications on other platforms.\n\nProject links\n-------------\n\n* `Documentation <https://py2app.readthedocs.io/en/latest/>`_\n\n* `Issue Tracker <https://github.com/ronaldoussoren/py2app/issues>`_\n\n* `Repository <https://github.com/ronaldoussoren/py2app/>`_\n\nRelease history\n===============\n\n\npy2app 0.22\n-----------\n\n* #300: Add support for ARM64 and Universal 2 binaries\n\n  .. note:: Support is highly experimental, these stubs have not been tested yet.\n\n* #299: Fix build error when building with the copy of Python 3 shipped\n  with Xcode.\n\n* #281: Generated bundle doesn\'t work on macOS 10.9 and 10.10.\n\npy2app 0.21\n-----------\n\n* PR 277 (Christian Clauss): Fix some Python 3 issues\n\n* #276: Rebuilt the binary stubs on a 10.12 machine to fix launching\n\npy2app 0.20\n-----------\n\n* Migrate to GitHub\n\n* #274: Fix an issue in the PyQt5 recipe\n\n* Fix issue with emulate-shell-environment option on macOS 10.15 (Catalina)\n\n* #269: Py2app didn\'t work with Python 3.8\n\npy2app 0.19\n-----------\n\n* #251: Add recipe for "botocore"\n\n* #253: "python setup.py py2app -A" creates invalid bundle from "venv" virtual environments\n\n* Updated recipe for PySide2 and new recipe for Shiboken2\n\n  Patch by Alberto Sottile.\n\npy2app 0.18\n-----------\n\n* #250: Add recipe for "six.moves", which also works when the six\n  library is vendored by other packages\n\npy2app 0.17\n-----------\n\n* #247: The new tkinter recipe didn\'t work properly for installations\n  that do use a framework install of Tcl/Tk.\n\npy2app 0.16\n-----------\n\n* #244: Copy the Tcl/Tk support libraries into the application bundle for\n  Python builds using a classic unix install of Tcl/Tk instead of a framework\n  build.\n\n  This results in working app bundles when a Python.org installation that\n  includes Tcl/Tk (such as Python 3.7).\n\n* Don\'t copy numpy into application just because the application uses\n  Pillow.\n\n* Add recipe for Pyside\n\n  Patch by Alberto Sottile\n\npy2app 0.15\n-----------\n\n* Fixed issues for Python 3.7, in particular changes in the plistlib library\n  (Issue #242, #239)\n\n* Updated dependencies on macholib, altgraph and modulegraph\n\n**Due to a bug in CPython 3.7.0 using -O does not work with that version of CPython**\n\npy2app 0.14.1\n-------------\n\n* Updated dependencies\n\n* Updated PyPI metadata\n\npy2app 0.14\n-----------\n\nFeatures:\n\n* Started using flake8 to improve coding style\n\nBug fixes:\n\n* Issue #222: The fix for issue #179 broke the argv emulator\n\n* Issue #226: Py2app could fail while reporting on possibly missing modules\n\n* Issue #228: The python executable included in the app bundle as ``sys.exectuable`` was not executable\n\n\npy2app 0.13\n-----------\n\nBug fixes:\n\n* Issue 185 in PyObjC\'s tracker: sysconfig using ``__import__`` in Python 3.6 or\n  later, which confuses modulegraph.\n\n* Pull request #17: Location of site-packages in the "--user" location has changed\n\n  Patch by Matt Mukerjee\n\nFeatures:\n\n* (None yet)\n\npy2app 0.12\n-----------\n\n* Pull request #15 by Armin Samii: Safer symlink and file copying\n\n* Update recipes: a number of recipe names conflicted with toplevel\n  modules imported by recipes. This causes problems on Python 2.7 (without\n  absolute imports)\n\npy2app 0.11\n-----------\n\n- Make sure the stdout/stderr streams of the main binary of the application\n  are unbuffered.\n\n  See `issue #177 in PyObjC\'s repository <https://github.com/ronaldoussoren/pyobjc/issues/177/on-python3-print-does-not-automatically>`_ for more information.\n\n- Fix issue #201: py2app is not compatible with pyvenv virtualenvs\n\n  With additional fix by Oskari Timperi.\n\n- Fix issue #179: the stdout/stderr streams are no longer forwarded to console.app using ASL (by default),\n  use "--redirect-stdout-to-asl" to enable the redirection functionality.\n\n  Note that for unclear reasons the redirection doesn\'t work on OSX 10.12 at the moment.\n\n- Fix issue #188: Troubles with lxml.isoschematron\n\n  The package \'lxml.isoschematron\' is not zip-safe and tries to load resources using the normal\n  filesystem APIs, which doesn\'t work when the package is part of a zipfile.\n\n- py2applet now longer uses "argv_emulation" by default, that results in too many problems.\n\n- Issue #174: clean up the summary about missing modules by removing warnings about things that aren\'t modules.\n\n  Also notes when an module is likely an alias for some other module. These changes should remove a lot\n  of false postive warnings from the output of py2app.\n\n- Fix issue #161: opengl recipe uses "file" function that isn\'t present on Python 3\n\n- Add "qt5" recipe that does the right thing for the PyQt5 wheel on PyPI (tested with PyQt5 5.6)\n\n- Add support for "@loader_path" in the link commands of C extension.\n\n  This makes it possible to use wheels that were processed by `delocate-listdeps <https://github.com/matthew-brett/delocate>`_\n  when building application bundles.\n\n- Do not report imports that are expected to be missing\n\n  Patch by Barry Scott.\n\npy2app 0.10\n-----------\n\n- The recipe for virtualenv calls a modulegraph method that was made\n  private in a recent release and hence no longer worked with py2app 0.9.\n\n  Update the recipe to work around this.\n\n\npy2app 0.9\n----------\n\n- issue #146, #147: The "python" binary in MyApp.app/Contents/MacOS was\n  the small stub exetable from framework builds, instead of the actual\n  command-line interpreter. The result is that you couldn\'t use\n  ``sys.executable`` to start a new interpreter, which (amongst others)\n  breaks multiprocessing.\n\n- pull request #7: Add support for PyQt5 to the sip recipe. Patch by\n  Mark Montague.\n\n- pull request #4: Copying PySide plugins was broken due to bad\n  indentation.\n\n- pull request #5: py2app was broken for python versions that\n  don\'t use _sysconfigdata.\n\n- issue #135: Don\'t sleep for a second after compiling a XIB file\n\n- issue #134: Remove target location before copying files into\n  the bundle.\n\n- issue #133: Ensure that the application\'s "Framework" folder\n  is on the search path for ``ctypes.util.find_library``.\n\n- issue #132: Depend on modulegraph 0.12 to avoid build errors\n  when the python code contains references to compatibility modules\n  that contain SyntaxErrors for the current python version.\n\n- Explicitly report modules that cannot be found at the end of\n  the run (for non-alias builds)\n\n  Note: This is just a warning, missing modules are not necessarily\n  a problem because modulegraph can detect imports for modules that\n  aren\'t used on OSX (for example)\n\n- Report modules that contain syntax errors at the end of\n  the run (for non-alias builds)\n\n  Note: This is just a warning, syntax errors be valid when the\n  dependency tree contains modules for the other major release\n  of python (e.g a compat_py2 module that contains compatibility\n  code for Python 2 and contains code that isn\'t valid Python 3)\n\npy2app 0.8.1\n------------\n\n- Loading scripts didn\'t work when --no-chdir was used\n\n  Reported by Barry Scott in private mail.\n\npy2app 0.8\n-----------\n\npy2app 0.8 is a feature release\n\n\n- Fixed argv emulator on OSX 10.9, the way the code detected that the application\n  was launched through the Finder didn\'t work on that OSX release.\n\n- The launcher binary is now linked with Cocoa, that should avoid some problems\n  with sandboxed applications (in particular: standard open panels don\'t seem\n  to work properly in a sandboxed application when the main binary is not\n  linked to AppKit)\n\n- Don\'t copy Python\'s Makefile, Setup file and the like into a bundle when\n  sysconfig and distutils.sysconfig don\'t need these files (basicly, when\n  using any recent python version).\n\n- Fix some issues with virtualenv support:\n\n  * detection of system installs of Python didn\'t work properly when using\n    a virtualenv. Because of this py2app did not create a "semi-standalone"\n    bundle when using a virtualenv created with /usr/bin/python.\n\n  * "semi-standalone" bundles created from a virtualenv included more files\n    when they should (in particular bits of the stdlib)\n\n- Issue #92: Add option \'--force-system-tk\' which ensures that the _tkinter\n  extension (used by Tkinter) is linked against the Apple build of Tcl/Tk,\n  even when it is linked to another framework in Python\'s std. library.\n\n  This will cause a build error when tkinter is linked with a major version of\n  Tcl/Tk that is not present in /System/Library/Frameworks.\n\n- Issue #80: Add support for copying system plugins into the application\n  bundle.\n\n  Py2app now supports a new option *include_plugins*. The value of this\n  is a list of paths to plugins that should be copied into the application\n  bundle.\n\n  Items in the list are either paths, or a tuple with the plugin type\n  and the path::\n\n      include_plugins=[\n        "MyPlugins/MyDocument.qlgenerator",\n        ("SystemConfiguration", "MyPlugins/MyConfig.plugin"),\n      ]\n\n  Py2app currently knows about the following plugin suffixes:\n  ``.qlgenerator``, ``.mdimporter``, ``.xpc``, ``.service``,\n  ``.prefPane``, ``.iaplugin`` and ``.action``. These plugins\n  can be added without specifying the plugin type.\n\n- Issue #83: Setup.py now refuses to install when the current\n  platform is not Mac OS X.\n\n  This makes it clear that the package is only supported on OSX and\n  avoids confusing errors later on.\n\n- Issue #39: It is now possible to have subpackages on\n  in the "packages" option of py2app.\n\n- Issue #37: Add recipe for pyEnchant\n\n  ..note::\n\n    The recipe only works for installations of pyEnchant\n    where pyEnchant is stored in the installation (such\n    as the binary eggs on PyPI), not for installations\n    that either use the "PYENCHANT_LIBRARY_PATH" environment\n    variable or MacPorts.\n\n- Issue #90: Removed the \'email\' recipe, but require a new enough version\n  of modulegraph instead. Because of this py2app now requires modulegraph\n  0.11 or later.\n\npy2app 0.7.4\n------------\n\n- Issue #77: the stdout/stderr streams of application and plugin bundles did not\n  end up in Console.app on OSX 10.8 (as they do on earlier releases of OSX). This\n  is due to a change in OSX.\n\n  With this version the application executable converts writes to the stdout\n  and stderr streams to the ASL logging subsystem with the options needed to\n  end up in the default view of Console.app.\n\n  NOTE: The stdout and stderr streams of plugin bundles are not redirected, as it\n  is rather bad form to change the global environment of the host application.\n\n- The i386, x86_64 and intel stub binaries are now compiled with clang on OSX 10.8,\n  instead of an older version of GCC. The other stub versions still are compiled\n  on OSX 10.6.\n\n- Issue #111: The site.py generated by py2app now contains a USER_SITE variable\n  (with a default value of ``None``) because some software tries to import the\n  value.\n\n- Py2app didn\'t preserve timestamps for files copied into application bundles,\n  and this can cause a bytecompiled file to appear older than the corresponding\n  source file (for packages copied in the bundle using the \'packages\' option).\n\n  Related to issue #101\n\n- Py2app also didn\'t copy file permissions for files copied into application\n  bundles, which isn\'t a problem in general but did cause binaries to lose\n  there executable permissions (as noted on Stackoverflow)\n\n- Issue #101: Set "PYTHONDONTWRITEBYTECODE" in the environment before\n  calling Py_Initialize to ensure that the interpreter won\'t try to\n  write bytecode files (which can cause problems when using sandboxed\n  applications).\n\n- Issue #105: py2app can now create app and plugin bundles when the main script\n  has an encoding other than ASCII, in particular for Python 3.\n\n- Issue #106: Ensure that the PIL recipe works on Python 3. PIL itself isn\'t\n  ported yet, but Pillow does work with Python 3.\n\n- "python setup.py install" now fails unless the machine is running Mac OS X.\n\n  I\'ve seen a number of reports of users that try to use py2app on Windows\n  or Linux to build OSX applications. That doesn\'t work, py2app now fails\n  during installation do make this clear.\n\n- Disabled the \'email\' recipe for python 3.x as it isn\'t needed there.\n\n- Issue #91: Added a recipe for `lxml <http://lxml.de/>`, needed because\n  lxml performs a number of imports from an extension and those cannot\n  be detected automaticly by modulegraph.\n\n- Issue #94: The site-packages zipfile in the application bundle now contains\n  zipfile entries for directories as well. This is needed to work around\n  a bug in the zipimporter for Python 3.3: it won\'t consider \'pkg/foo.py\' to be\n  in namespace package \'pkg\' unless there is a zipfile entry for the \'pkg\'\n  folder (or there is a \'pkg/__init__.py\' entry).\n\n- Issue #97: Fixes a problem with the pyside and sip recipes when the \'qt_plugins\'\n  option is used for \'image_plugins\'.\n\n- Issue #96: py2app should work with python 2.6 again (previous releases didn\'t\n  work due to using the sysconfig module introduced in python 2.7)\n\n- Issue #99: appstore requires a number of symlinks in embedded frameworks.\n\n  (Version 0.7 already added a link Python.frameworks/Versions/Current, this\n  versions also adds Python.framework/Python and Python.framework/Resources with\n  the value required by the appstore upload tool).\n\n- Py2app copied stdlib packages into the app bundle for semi-standalone builds\n  when they are mentioned in the \'--packages\' option (either explicitly or\n  by a recipe). This was unintentional, semi-standlone builds should rely on\n  the external Python framework for the stdlib.\n\n  .. note::\n\n     Because of this bug parts of the stdlib of ``/usr/bin/python`` could be\n     copied into app bundles created with py2app.\n\npy2app 0.7.3\n------------\n\npy2app 0.7.3 is a bugfix release\n\n- Issue #82: Remove debug print statement from py2app.util.LOADER that\n  caused problems with Python 3.\n\n- Issue #81: Py2app now fails with an error when trying to build a bundle\n  for a unix-style shared library build of Python (``--enable-shared``) unless\n  you are using a recent enough patchlevel of python (2.7.4, 3.2.3, 3.3.1,\n  3.4.0, all of them are not released yet).\n\n  The build failure was added to avoid a very confusing error when trying\n  to start the generated application due to a bug in the way python reads\n  the environment (for shared library builds on Mac OS X).\n\n- Py2app will also give an error message when the python binary does not\n  have a shared library (or framework) at all.\n\n- Issue #87: Ignore \'.git\' and \'.hg\' directories while copying package data\n  (\'.svn\' and \'CVS\' were already ignored).\n\n- Issue #65: the fix in 0.7 to avoid copying a symlinked library twice caused\n  problems for some users because only one of the file names ended up in the\n  application bundle. This release ensures that both names exist (one as a\n  symbolic name to the other).\n\n- Issue #88: Ensure that the fix for #65 won\'t try to create a symlink that\n  points to itself. This could for example occur with homebrew, where the\n  exposed lib directory contains symlinks to a cellar, while tye install_name\n  does mention the "public" lib directory::\n\n     $ ls -l /opt/homebrew/lib\n     ...\n     libglib-2.0.0.dylib -> ../Cellar/glib/2.32.4/lib/libglib-2.0.0.dylib\n     ...\n\n     $ otool -vL /opt/homebrew/lib/libglib-2.0.0.dylib\n     /opt/homebrew/lib/libglib-2.0.0.dylib:\n        /opt/homebrew/lib/libglib-2.0.0.dylib (compatibility version 3201.0.0, current version 3201.4.0)\n        time stamp 1 Thu Jan  1 01:00:01 1970\n     ...\n\n\n\npy2app 0.7.2\n------------\n\npy2app 0.7.2 is a bugfix release\n\n- Issue #75: Don\'t remove ``--dist-dir``, but only remove the old version\n  of the objects we\'re trying to build (if that exists).\n\n  This once again makes it possible to have a number of setup.py files that\n  build plugins into the same target folder (such as the plugins folder\n  of an application)\n\n- Issue #78: Packages added using the ``--packages`` option didn\'t end up\n  on ``sys.path`` for semi-standalone applications.\n\n  Reported by Steve Strassmann\n\n- Issue #76: Semi-standalone packages using extensions modules coudn\'t use\n  extensions unless they also used the ``--site-packages`` option (and\n  the extensions are in the site-packages directory).\n\n  Fixes some problems with PyQt and wxWidgets when using the system installation\n  of Python.\n\n  Patch by Dan Horner.\n\n- It is currently not possible to use a subpackage ("foo.bar") in the list\n  of packages for the "packages" option. Py2app now explicitly checks for this\n  and prints an error message instead of building an application that doesn\'t\n  work.\n\n  Issue: #39\n\n\npy2app 0.7.1\n------------\n\npy2app 0.7.1 is a bugfix release\n\n- Always include \'pkg_resources\', this is needed to correctly work with\n  setuptools namespace packages, the __init__.py files of those contain\n  ``__import__(\'pkg_resources\')`` and that call isn\'t recognized as an import\n  by the bytecode scanner.\n\n- Issue #67: py2applet didn\'t work with python 3 due to the use of \'raw_input\'\n\n  Reported by Andrew Barnert.\n\n- Issue #68: the "extra-scripts" feature introduced in 0.7 couldn\'t copy scripts\n  that aren\'t in the same directory as "setup.py".\n\n  Reported by Andrew Barnert.\n\n- For semi-standalone applications the "lib-dynload" directory inside the\n  application was not on "sys.path", which resulted in launch failures\n  when using an extension that is not in the stdlib.\n\n- Issue #70: application fails to launch when script uses Windows line endings\n\n  Reported by Luc Jean.\n\npy2app 0.7\n------------\n\npy2app 0.7 is a bugfix release\n\n- Issue #65: generated bundle would crash when two libraries linked to the\n  same library using different names (one refering to the real name, the other\n  to a symlink).\n\n  An example if this is an application using wxWidgets when wxWidgets is installed\n  using homebrew.\n\n  Reported by "Bouke".\n\n- Issue #13: It is now possible to add helper scripts to a bundle, for\n  example for creating a GUI that starts a helper script in the background.\n\n  This can be done by using the option "--extra-scripts", the value of which is a list\n  of script files (".py" or ".pyw" files).\n\n- Smarter matplotlib recipe, it is now possible to specify which backends should\n  be included. Issue #44, reported by Adam Kovics.\n\n  The argument to ``--matplotlib-backends`` (or \'matplotlib_backends\' in setup.py)\n  is a list of plugins to include. Use \'-\' to not include backends other than those\n  found by the import statement analysis, and \'*\' to include all backends (without\n  necessarily including all of matplotlib)\n\n  As an example, use ``--matplotlib-backends=wxagg`` to include just the wxagg\n  backend.\n\n  Default is to include the entire matplotlib package.\n\n- The packages included by a py2app recipe weren\'t processed by modulegraph and\n  hence their dependencies were not always included.\n\n- Fix virtualenv support: alias builds in a virtual environment failed to work.\n\n  (There are still issues with semi-standalone and alias plugin bundles in\n  a virtualenv environment).\n\n- issue #18: improved PyQt and PySide support.\n\n  Py2app now has a new option named "--qt-plugins" (or "qt_plugins" in setup.py),\n  this option specify a list of plugins that should be included in the\n  application bundle. The items of the list can have a number of forms:\n\n  * "plugintype/libplugin.dylib"\n\n    Specify one particular plugin\n\n  * "plugintype/\\*foo\\*"\n\n    Specify one or more plugins using a glob pattern\n\n  * "plugintype"\n\n    Include all plugins of a type, equivalent to "plugintype/\\*".\n\n  The plugins are copied into "Resources/qt_plugins" and py2app adds a "qt.conf"\n  file that points to that location for plugins.\n\n- issue #49: package data that is a zipfile is now correctly copied into\n  the bundle instead of extracting the archive.\n\n- issue #59: compile site.py to ensure that the generated bundle doesn\'t\n  change on first run.\n\n  This is nice to have in general, and essential when using code signing\n  because the signature will break when a new file is added after signing.\n\n  Reported by Michael McCracken.\n\n- issue #60: recipe for "email" package was not loaded\n\n  Reported by Chris Beaumont\n\n- issue #46: py2app no longer warns about the Qt license. We don\'t warn about\n  other possibly GPL licensed software either and py2app is not\n  a license-enforcement tool.\n\n  Reported by briank_in_la.\n\n- Generated bundles always started with python optimization active\n  (that is, as if running as \'python -O\').\n\n- Fix issue #53: py2app would crash if a data file happened to\n  be a zipfile.\n\n- py2app copies data files in the directory for a package into\n  the application bundle. It also did this for directories that\n  represent subpackages, which made it impossible to exclude\n  subpackages.\n\n- added recipe for wxPython because some subpackages of wxPython\n  use ``__path__`` trickery that confuses modulegraph.\n\n- recipes can now return a list of additional entries for the\n  \'includes\' list.\n\n- rewritten the recipe for matplotlib. The recipe no longer includes\n  the entire package, but just the "mpl-data" directory.\n\n  WARNING: This recipe has had limited testing.\n\n- fix mixed indentation (tabs and spaces) in argv_emulation.py,\n  which caused installation failures on python 3.x (issue #40)\n\n- Issue #43: py2app now creates a symlink named "Current" in the\n  \'Versions\' directory of the embedded Python framework to comply\n  with a requirement for the Mac App-store.\n\n- on some OSX releases the application receives both the\n  "open application" and "open documents" Apple Events during startup,\n  which broke an assumption in argv_emulation.py.\n\n- py2app is more strict w.r.t. explictly closing files, this avoids\n  ResourceWarnings for unclosed files.\n\n- fix test issue with semi-standalone builds on Python 3.2\n\n- added recipe for pyzmq\n\n- Don\'t use the version information from Python.framework\'s Info.plist,\n  but use ``sys.version_info``. This fixes a build problem with EPD.\n\n- Ignore some more files when copying package data:\n\n  - VIM swap files (``.foo.py.swp``)\n\n  - Backup files for a number of tools: ``foo.orig`` and ``foo~``\n\npy2app 0.6.4\n------------\n\npy2app 0.6.4 is a bugfix and minor feature release\n\n- Issue #28: the argv emulator crashes in 64-bit mode on OSX 10.5\n\n  Fixing this issue required yet another rewrite of the argv_emulator\n  code.\n\n- Added option \'--arch=VALUE\' which can be used to select the set of\n  architectures for the main executable. This defaults to the set of\n  architectures supported by the python interpreter and can be used to\n  drop support for some architectures (for example when you\'re using a\n  python binary that supports both 32-bit and 64-bit code and use a\n  GUI library that does not yet work in 64-bit mode).\n\n  Valid values for the argument are archectures used in the list below\n  and the following groups of architectures:\n\n  * fat:        i386, ppc\n\n  * fat3:       i386, x86_64, ppc\n\n  * univeral:   i386, x86_64, ppc, ppc64\n\n  * intel:      i386, x86_64\n\n\n\n- Issue #32: fix crash when application uses PySide\n\n  This is partially fixed in macholib (release 1.4.3)\n\n- The \'-O\' flag of py2app now defaults to the python optimization level\n  when using python 2.6 or later.\n\n- Issue #31: honor optimze flag at runtime.\n\n  Until now an application bundle created by py2app would also run without\n  the "-O" flag, even when the user specified it should. This is now fixed.\n\n- Issue #33: py2app\'s application bundle launcher now clears the environment\n  variable ``PYOBJC_BUNDLE_ADDRESS``, avoids a crash when using PyObjC in an\n  application launched from a py2app based plugin bundle.\n\n- py2app\'s bundle launcher set the environment variable ``PYOBJC_BUNDLE_ADDRESS``,\n  this variable is now deprecated. Use ``PYOBJC_BUNDLE_ADDRESS<PID>`` instead\n  (replace ``<PID>`` by the process ID of the current process).\n\n- When using the system python we now explicitly add Apple\'s additional packages\n  (like PyObjC and Twisted) to ``sys.path``.\n\n  This fixes and issue reported by Sean Robinson: py2app used to create a non-working\n  bundle when you used these packages because the packages didn\'t get included\n  (as intented), but were not available on ``sys.path`` either.\n\n- Fixed the recipe for sip, which in turn ensures that PyQt4 applications\n  work.\n\n  As before the SIP recipe is rather crude, it will include *all* SIP-based\n  packages into your application bundle when it detects a module that uses\n  SIP.\n\n- The \'Resources\' folder is no longer on the python search path,\n  it contains the scripts while Python modules and packages are located\n  in the site-packages directory. This change is related to issue #30.\n\n- The folder \'Resources/Python/site-packages\' is no longer on the python\n  search path. This folder is not used by py2app itself, but might by\n  used by custom build scripts that wrap around py2app.\n\n- Issue #30: py2app bundles failed to launch properly when the scriptfile\n  has the same name as a python package used by the application.\n\n- Issue #15: py2app now has an option to emulate the shell environment you\n  get by opening a window in the Terminal.\n\n  Usage: ``python setup.py py2app --emulate-shell-environment``\n\n  This option is experimental, it is far from certain that the implementation\n  works on all systems.\n\n- Issue #16: ``--argv-emulation`` now works with Python 3.x and in 64-bit\n  executables.\n\n- Issue #17: py2applet script defaults \'argv_emulation\' to False when your using\n  a 64-bit build of python, because that option is not supported on\n  such builds.\n\n- py2app now clears the temporary directory in \'build\' and the output directory\n  in \'dist\' before doing anything. This avoids unwanted interactions between\n  results from a previous builds and the current build.\n\n- Issue #22: py2app will give an error when the specified version is invalid,\n  instead of causing a crash in the generated executable.\n\n- Issue #23: py2app failed to work when an .egg directory was implictly added\n  to ``sys.path`` by setuptools and the "-O" option was used (for example\n  ``python setup.py py2app -O2``)\n\n- Issue #26: py2app copied the wrong executable into the application bundle\n  when using virtualenv with a framework build of Python.\n\npy2app 0.6.3\n------------\n\npy2app 0.6.3 is a bugfix release\n\n- py2app failed to compile .xib files\n  (as reported on the pythonmac-sig mail-ing list).\n\n\npy2app 0.6.2\n------------\n\npy2app 0.6.2 is a bugfix release\n\n- py2app failed to copy the iconfile into application bundle\n  (reported by Russel Owen)\n\n- py2app failed to copy resources and data files as well\n  (the ``resource`` key in the py2ap options dictionary and\n  the ``data_files`` argument to the setup function).\n\n  Issue #19, reported by bryon(at)spideroak.com.\n\n- py2app failed to build application bundles when using virtualenv\n  due to assumptions about the relation between ``sys.prefix`` and\n  ``sys.executable``.\n\n  Report and fix by Erik van Zijst.\n\n- Ensure that the \'examples\' directory is included in the source\n  archive\n\npy2app 0.6.1\n------------\n\npy2app 0.6.1 is a bugfix release\n\nBugfixes:\n\n- py2app failed to build the bundle when python package contained\n  a zipfile with data.\n\n  This version solves most of that problem using a rough\n  workaround (the issue is fixed when the filename ends with \'.zip\').\n\n- The code that recreates the stub executables when they are\n  older than the source code now uses ``xcode-select`` to\n  find the root of SDKs.\n\n  This makes it possible to recreate these executables on machines\n  where both Xcode 3 and Xcode 4 are installed and Xcode 3 is\n  the default Xcode.\n\n- The stub executables were regenerated using Xcode 3\n\n  As a word of warning: Xcode 4 cannot be used to rebuild the\n  stub executables, in particular not those that have support\n  for the PPC architecture.\n\n- Don\'t rebuild the stub executables automaticly, that\'s\n  unsafe with Xcode 4 and could trigger accidently when\n  files are installed in a different order than expected.\n\n- Small tweaks to the testsuite to ensure that they work\n  on systems with both Xcode3 and Xcode4 (Xcode3 must be\n  the selected version).\n\n- Better cleanup in the testsuite when ``setupClass`` fails.\n\npy2app 0.6\n----------\n\npy2app 0.6 is a minor feature release\n\n\nFeatures:\n\n- it is now possible to specify which python distributions must\n  be availble when building the bundle by using the\n  "install_requires" argument of the ``setup()`` function::\n\n     setup(\n\n         ...\n\t install_requires = [\n\t \t"pyobjc == 2.2"\n\t ],\n     )\n\n- py2app can now package namespace packages that were installed\n  using `pip <http://pypi.python.org/pypi/pip>` or the\n  setuptools install option ``--single-version-externally-managed``.\n\n- the bundle template now supports python3, based on a patch\n  by Virgil Dupras.\n\n- alias builds no longer use Carbon Aliases and therefore are\n  supported with python3 as well (patch by Virgil Dupras)\n\n- argv emulation doesn\'t work in python 3, this release\n  will tell you abou this instead of silently failing to\n  build a working bundle.\n\n- add support for custom URLs to the argv emulation code\n  (patch by Brendan Simon).\n\n  You will have to add a "CFBundleURLTypes" key to your Info.plist to\n  use this, the argv emulation code will ensure that the URL\n  to open will end up in ``sys.argv``.\n\n- ``py2app.util`` contains a number of functions that are now\n  deprecated an will be removed in a future version, specifically:\n  ``os_path_islink``, ``os_path_isdir``, ``path_to_zip``,\n  ``get_zip_data``, ``get_mtime``,  and ``os_readlink``.\n\n- The module ``py2app.simpleio`` no longer exists, and should never\n  have been in the repository (it was part of a failed rewrite of\n  the I/O layer).\n\nBug fixes:\n\n- fix problem with symlinks in copied framework, as reported\n  by Dan Ross.\n\n- py2applet didn\'t work in python 3.x.\n\n- The ``--alias`` option didn\'t work when building a plugin\n  bundle (issue #10, fix by Virgil Dupras)\n\n- Avoid copying the __pycache__ directory in python versions\n  that implement PEP 3147 (Python 3.2 and later)\n\n- App bundles with Python 3 now work when the application is\n  stored in a directory with non-ASCII characters in the full\n  name.\n\n- Do not compile ``.nib`` files, it is not strictly needed and\n  breaks PyObjC projects that still use the NibClassBuilder code.\n\n- Better error messsages when trying to include a non-existing\n  file as a resource.\n\n- Don\'t drop into PDB when an exception occurs.\n\n- Issue #5: Avoid a possible stack overflow in the bundle executable\n\n- Issue #9: Work with python 3.2\n\n- Fix build issues with python 2.5 (due to usage of too modern distutils\n  command subclasses)\n\n- The source distribution didn\'t include all files that needed to be\n  it ever since switching to mercurial, I\'ve added a MANIFEST.in\n  file rather than relying on setuptool\'s autoguessing of files to include.\n\n- Bundle template works again with semi-standalone builds (such as\n  when using a system python), this rewrites the fix for issue #10\n  mentioned earlier.\n\n- Ensure py2app works correctly when the sources are located in a\n  directory with non-ascii characters in its name.\n\n\npy2app 0.5.2\n------------\n\npy2app 0.5.2 is a bugfix release\n\nBug fixes:\n\n- Ensure that the right stub executable gets found when using\n  the system python 2.5\n\npy2app 0.5.1\n------------\n\npy2app 0.5.1 is a bugfix release\n\nBug fixes:\n\n- Ensure stub executables get included in the egg files\n\n- Fix name of the bundletemplate stub executable for 32-bit builds\n\n\n\npy2app 0.5\n----------\n\npy2app 0.5 is a minor feature release.\n\nFeatures:\n\n- Add support for the ``--with-framework-name`` option of Python\'s\n  configure script, that is: py2app now also works when the Python\n  framework is not named \'Python.framework\'.\n\n- Add support for various build flavours of Python (32bit, 3-way, ...)\n\n- py2app now actually works for me (ronaldoussoren@mac.com) with a\n  python interpreter in a virtualenv environment.\n\n- Experimental support for python 3\n\nBug fixes:\n\n- Fix recipe for matplotlib: that recipe caused an exception with\n  current versions of matplotlib and pytz.\n\n- Use modern API\'s in the alias-build bootstrap code, without\n  this \'py2app -A\' will result in broken bundles on a 64-bit build\n  of Python.\n  (Patch contributed by James R Eagan)\n\n- Try both \'import Image\' and \'from PIL import Image\' in the PIL\n  recipe.\n  (Patch contributed by Christopher Barker)\n\n- The stub executable now works for 64-bit application bundles\n\n- (Lowlevel) The application stub was rewritten to use\n  ``dlopen`` instead of ``dyld`` APIs. This removes deprecation\n  warnings during compilation.\n\npy2app 0.4.3\n------------\n\npy2app 0.4.3 is a bugfix release\n\nBug fixes:\n\n- A bad format string in build_app.py made it impossible to copy the\n  Python framework into an app bundle.\n\npy2app 0.4.2\n------------\n\npy2app 0.4.2 is a minor feature release\n\nFeatures:\n\n- When the \'--strip\' option is specified we now also remove \'.dSYM\'\n  directories from the bundle.\n\n- Remove dependency on a \'version.plist\' file in the python framework\n\n- A new recipe for `PyQt`_ 4.x. This recipe was donated by Kevin Walzer.\n\n- A new recipe for `virtualenv`_, this allows you to use py2app from\n  a virtual environment.\n\n.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv\n\n- Adds support for converting ``.xib`` files (NIB files for\n  Interface Builder 3)\n\n- Introduces an experimental plugin API for data converters.\n\n  A conversion plugin should be defined as an entry-point in the\n  ``py2app.converter`` group::\n\n       setup(\n         ...\n\t entry_points = {\n\t\t \'py2app.converter\': [\n\t\t     "label          = some_module:converter_function",\n\t\t  ]\n\t  },\n\t  ...\n      )\n\n  The conversion function should be defined like this::\n\n      from py2app.decorators import converts\n\n      @converts(\'.png\')\n      def optimze_png(source, proposed_destionation, dryrun=0):\n         # Copy \'source\' to \'proposed_destination\'\n\t # The conversion is allowed to change the proposed\n\t # destination to another name in the same directory.\n         pass\n\n.. `virtualenv`_: http://pypi.python.org/pypi/virtualenv\n\nBuf fixes:\n\n- This fixes an issue with copying a different version of Python over\n  to an app/plugin bundle than the one used to run py2app with.\n\n\npy2app 0.4.0\n------------\n\npy2app 0.4.0 is a minor feature release (and was never formally released).\n\nFeatures:\n\n- Support for CoreData mapping models (introduced in Mac OS X 10.5)\n\n- Support for python packages that are stored in zipfiles (such as ``zip_safe``\n  python eggs).\n\nBug fixes:\n\n- Fix incorrect symlink target creation with an alias bundle that has included\n  frameworks.\n\n- Stuffit tends to extract archives recursively, which results in unzipped\n  code archives inside py2app-created bundles. This version has a workaround\n  for this "feature" for Stuffit.\n\n- Be more carefull about passing non-constant strings as the template argumenti\n  of string formatting functions (in the app and bundle templates), to avoid\n  crashes under some conditions.\n\npy2app 0.3.6\n------------\n\npy2app 0.3.6 is a minor bugfix release.\n\nBug fixes:\n\n- Ensure that custom icons are copied into the output bundle\n\n- Solve compatibility problem with some haxies and inputmanager plugins\n\n\npy2app 0.3.5\n------------\n\npy2app 0.3.5 is a minor bugfix release.\n\nBug fixes:\n\n- Resolve disable_linecache issue\n\n- Fix Info.plist and Python path for plugins\n\n\npy2app 0.3.4\n------------\n\npy2app 0.3.4 is a minor bugfix release.\n\nBug fixes:\n\n- Fixed a typo in the py2applet script\n\n- Removed some, but not all, compiler warnings from the bundle template\n  (which is still probably broken anyway)\n\n\npy2app 0.3.3\n------------\n\npy2app 0.3.3 is a minor bugfix release.\n\nBug Fixes:\n\n- Fixed a typo in the argv emulation code\n\n- Removed the unnecessary py2app.install hack (setuptools does that already)\n\n\npy2app 0.3.2\n------------\n\npy2app 0.3.2 is a major bugfix release.\n\nFunctional changes:\n\n- Massively updated documentation\n\n- New prefer-ppc option\n\n- New recipes: numpy, scipy, matplotlib\n\n- Updated py2applet script to take options, provide --make-setup\n\nBug Fixes:\n\n- No longer defaults to LSPrefersPPC\n\n- Replaced stdlib usage of argvemulator to inline version for i386\n  compatibility\n\n\npy2app 0.3.1\n------------\n\npy2app 0.3.1 is a minor bugfix release.\n\nFunctional changes:\n\n- New EggInstaller example\n\nBug Fixes:\n\n- Now ensures that the executable is +x (when installed from egg this may not\n  be the case)\n\n\npy2app 0.3.0\n------------\n\npy2app 0.3.0 is a major feature enhancements release.\n\nFunctional changes:\n\n- New --xref (-x) option similar to py2exe\'s that produces\n  a list of modules and their interdependencies as a HTML\n  file\n\n- sys.executable now points to a regular Python interpreter\n  alongside the regular executable, so spawning sub-interpreters\n  should work much more reliably\n\n- Application bootstrap now detects paths containing ":"\n  and will provide a "friendly" error message instead of just\n  crashing <http://python.org/sf/1507224>.\n\n- Application bootstrap now sets PYTHONHOME instead of\n  a large PYTHONPATH\n\n- Application bootstrap rewritten in C that links to\n  CoreFoundation and Cocoa dynamically as needed,\n  so it doesn\'t imply any particular version of the runtime.\n\n- Documentation and examples changed to use setuptools\n  instead of distutils.core, which removes the need for\n  the py2app import\n\n- Refactored to use setuptools, distributed as an egg.\n\n- macholib, bdist_mpkg, modulegraph, and altgraph are now\n  separately maintained packages available on PyPI as eggs\n\n- macholib now supports little endian architectures,\n  64-bit Mach-O headers, and reading/writing of\n  multiple headers per file (fat / universal binaries)\n\n\npy2app 0.2.1\n------------\n\npy2app 0.2.1 is a minor bug fix release.\n\nBug Fixes:\n\n- macholib.util.in_system_path understands SDKs now\n\n- DYLD_LIBRARY_PATH searching is fixed\n\n- Frameworks and excludes options should work again.\n\n\npy2app 0.2.0\n------------\n\npy2app 0.2.0 is a minor bug fix release.\n\nFunctional changes:\n\n- New datamodels option to support CoreData.  Compiles\n  .xcdatamodel files and places them in the Resources dir\n  (as .mom).\n\n- New use-pythonpath option.  The py2app application bootstrap\n  will no longer use entries from PYTHONPATH unless this option\n  is used.\n\n- py2app now persists information about the build environment\n  (python version, executable, build style, etc.) in the\n  Info.plist and will clean the executable before rebuilding\n  if anything at all has changed.\n\n- bdist_mpkg now builds packages with the full platform info,\n  so that installing a package for one platform combination\n  will not look like an upgrade to another platform combination.\n\nBug Fixes:\n\n- Fixed a bug in standalone building, where a rebuild could\n  cause an unlaunchable executable.\n\n- Plugin bootstrap should compile/link correctly\n  with gcc 4.\n\n- Plugin bootstrap no longer sets PYTHONHOME and will\n  restore PYTHONPATH after initialization.\n\n- Plugin bootstrap swaps out thread state upon plug-in\n  load if it is the first to initialize Python.  This\n  fixes threading issues.\n\npy2app 0.1.9\n------------\n\npy2app 0.1.9 is a minor bug fix release.\n\nBugs fixed:\n\n- bdist_mpkg now builds zip files that are correctly unzipped\n  by all known tools.\n\n- The behavior of the bootstrap has changed slightly such that\n  ``__file__`` should now point to your main script, rather than\n  the bootstrap.  The main script has also moved to ``Resources``,\n  from ``Resources/Python``, so that ``__file__`` relative resource\n  paths should still work.\n\npy2app 0.1.8\n------------\n\npy2app 0.1.8 is a major enhancements release:\n\nBugs fixed:\n\n- Symlinks in included frameworks should be preserved correctly\n  (fixes Tcl/Tk)\n\n- Fixes some minor issues with alias bundles\n\n- Removed implicit SpiderImagePlugin -> ImageTk reference in PIL\n  recipe\n\n- The ``--optimize`` option should work now\n\n- ``weakref`` is now included by default\n\n- ``anydbm``\'s dynamic dependencies are now in the standard implies\n  list\n\n- Errors on app launch are brought to the front so the user does\n  not miss them\n\n- bdist_mpkg now compatible with pychecker (data_files had issues)\n\nOptions changed:\n\n- deprecated ``--strip``, it is now on by default\n\n- new ``--no-strip`` option to turn off stripping of executables\n\nNew features:\n\n- Looks for a hacked version of the PyOpenGL __init__.py so that\n  it doesn\'t have to include the whole package in order to get\n  at the stupid version file.\n\n- New ``loader_files`` key that a recipe can return in order to\n  ensure that non-code ends up in the .zip (the pygame recipe\n  uses this)\n\n- Now scans all files in the bundle and normalizes Mach-O load\n  commands, not just extensions.  This helps out when using the\n  ``--package`` option, when including frameworks that have plugins,\n  etc.\n\n- An embedded Python interpreter is now included in the executable\n  bundle (``sys.executable`` points to it), this currently only\n  works for framework builds of Python\n\n- New ``macho_standalone`` tool\n\n- New ``macho_find`` tool\n\n- Major enhancements to the way plugins are built\n\n- bdist_mpkg now has a ``--zipdist`` option to build zip files\n  from the built package\n\n- The bdist_mpkg "Installed to:" description is now based on the\n  package install root, rather than the build root\n\npy2app 0.1.7\n------------\n\n`py2app`_ 0.1.7 is a bug fix release:\n\n- The ``bdist_mpkg`` script will now set up sys.path properly, for setup scripts\n  that require local imports.\n\n- ``bdist_mpkg`` will now correctly accept ``ReadMe``, ``License``, ``Welcome``,\n  and ``background`` files by parameter.\n\n- ``bdist_mpkg`` can now display a custom background again (0.1.6 broke this).\n\n- ``bdist_mpkg`` now accepts a ``build-base=`` argument, to put build files in\n  an alternate location.\n\n- ``py2app`` will now accept main scripts with a ``.pyw`` extension.\n\n- ``py2app``\'s not_stdlib_filter will now ignore a ``site-python`` directory as\n  well as ``site-packages``.\n\n- ``py2app``\'s plugin bundle template no longer displays GUI dialogs by default,\n  but still links to ``AppKit``.\n\n- ``py2app`` now ensures that the directory of the main script is now added to\n  ``sys.path`` when scanning modules.\n\n- The ``py2app`` build command has been refactored such that it would be easier\n  to change its behavior by subclassing.\n\n- ``py2app`` alias bundles can now cope with editors that do atomic saves\n  (write new file, swap names with existing file).\n\n- ``macholib`` now has minimal support for fat binaries.  It still assumes big\n  endian and will not make any changes to a little endian header.\n\n- Add a warning message when using the ``install`` command rather than installing\n  from a package.\n\n- New ``simple/structured`` example that shows how you could package an\n  application that is organized into several folders.\n\n- New ``PyObjC/pbplugin`` Xcode Plug-In example.\n\npy2app 0.1.6\n------------\n\nSince I have been slacking and the last announcement was for 0.1.4, here are the\nchanges for the soft-launched releases 0.1.5 and 0.1.6:\n\n`py2app`_ 0.1.6 was a major feature enhancements release:\n\n- ``py2applet`` and ``bdist_mpkg`` scripts have been moved to Python modules\n  so that the functionality can be shared with the tools.\n\n- Generic graph-related functionality from ``py2app`` was moved to\n  ``altgraph.ObjectGraph`` and ``altgraph.GraphUtil``.\n\n- ``bdist_mpkg`` now outputs more specific plist requirements\n  (for future compatibility).\n\n- ``py2app`` can now create plugin bundles (MH_BUNDLE) as well as executables.\n  New recipe for supporting extensions built with `sip`_, such as `PyQt`_.  Note that\n  due to the way that `sip`_ works, when one sip-based extension is used, *all*\n  sip-based extensions are included in your application.  In practice, this means\n  anything provided by `Riverbank`_, I don\'t think anyone else uses `sip`_ (publicly).\n\n- New recipe for `PyOpenGL`_.  This is very naive and simply includes the whole\n  thing, rather than trying to monkeypatch their brain-dead\n  version acquisition routine in ``__init__``.\n\n- Bootstrap now sets ``ARGVZERO`` and ``EXECUTABLEPATH`` environment variables,\n  corresponding to the ``argv[0]`` and the ``_NSGetExecutablePath(...)`` that the\n  bundle saw.  This is only really useful if you need to relaunch your own\n  application.\n\n- More correct ``dyld`` search behavior.\n\n- Refactored ``macholib`` to use ``altgraph``, can now generate `GraphViz`_ graphs\n  and more complex analysis of dependencies can be done.\n\n- ``macholib`` was refactored to be easier to maintain, and the structure handling\n  has been optimized a bit.\n\n- The few tests that there are were refactored in `py.test`_ style.\n\n- New `PyQt`_ example.\n\n- New `PyOpenGL`_ example.\n\n\nSee also:\n\n- http://mail.python.org/pipermail/pythonmac-sig/2004-December/012272.html\n\n.. _`py.test`: http://codespeak.net/py/current/doc/test.html\n.. _`PyOpenGL`: http://pyopengl.sourceforge.net/\n.. _`Riverbank`: http://www.riverbankcomputing.co.uk/\n.. _`sip`: http://www.riverbankcomputing.co.uk/sip/index.php\n.. _`PyQt`: http://www.riverbankcomputing.co.uk/pyqt/index.php\n.. _`docutils`: http://docutils.sf.net/\n.. _`setuptools`: http://cvs.eby-sarna.com/PEAK/setuptools/\n\npy2app 0.1.5\n------------\n\n`py2app`_ 0.1.5 is a major feature enhancements release:\n\n- Added a ``bdist_mpkg`` distutils extension, for creating Installer\n  an metapackage from any distutils script.\n\n  - Includes PackageInstaller tool\n\n  - bdist_mpkg script\n\n  - setup.py enhancements to support bdist_mpkg functionality\n\n- Added a ``PackageInstaller`` tool, a droplet that performs the same function\n    as the ``bdist_mpkg`` script.\n\n- Create a custom ``bdist_mpkg`` subclass for `py2app`_\'s setup script.\n\n- Source package now includes `PJE`_\'s `setuptools`_ extension to distutils.\n\n- Added lots of metadata to the setup script.\n\n- ``py2app.modulegraph`` is now a top-level package, ``modulegraph``.\n\n- ``py2app.find_modules`` is now ``modulegraph.find_modules``.\n\n- Should now correctly handle paths (and application names) with unicode characters\n  in them.\n\n- New ``--strip`` option for ``py2app`` build command, strips all Mach-O files\n  in output application bundle.\n\n- New ``--bdist-base=`` opt
Dec 10 13:59:42 search0 swh[3329]: ion for ``py2app`` build command, allows an alternate\n  build directory to be specified.\n\n- New `docutils`_ recipe.\n  Support for non-framework Python, such as the one provided by `DarwinPorts`_.\n\nSee also:\n\n- http://mail.python.org/pipermail/pythonmac-sig/2004-October/011933.html\n\n.. _`py.test`: http://codespeak.net/py/current/doc/test.html\n.. _`GraphViz`: http://www.pixelglow.com/graphviz/\n.. _`PyOpenGL`: http://pyopengl.sourceforge.net/\n.. _`Riverbank`: http://www.riverbankcomputing.co.uk/\n.. _`sip`: http://www.riverbankcomputing.co.uk/sip/index.php\n.. _`PyQt`: http://www.riverbankcomputing.co.uk/pyqt/index.php\n.. _`DarwinPorts`: http://darwinports.opendarwin.org/\n.. _`setuptools`: http://cvs.eby-sarna.com/PEAK/setuptools/\n.. _`PJE`: http://dirtSimple.org/\n.. _`PyObjC`: http://pyobjc.sourceforge.net/\n\npy2app 0.1.4\n------------\n\n`py2app`_ 0.1.4 is a minor bugfix release:\n\n- The ``altgraph`` from 0.1.3 had a pretty nasty bug in it that prevented\n  filtering from working properly, so I fixed it and bumped to 0.1.4.\n\npy2app 0.1.3\n------------\n\n`py2app`_ 0.1.3 is a refactoring and new features release:\n\n- ``altgraph``, my fork of Istvan Albert\'s `graphlib`_, is now part of the\n  distribution\n\n- ``py2app.modulegraph`` has been refactored to use ``altgraph``\n\n- `py2app`_ can now create `GraphViz`_ DOT graphs with the ``-g`` option\n  (`TinyTinyEdit example`_)\n\n- Moved the filter stack into ``py2app.modulegraph``\n\n- Fixed a bug that may have been in 0.1.2 where explicitly included packages\n  would not be scanned by ``macholib``\n\n- ``py2app.apptemplate`` now contains a stripped down ``site`` module as\n  opposed to a ``sitecustomize``\n\n- Alias builds are now the only ones that contain the system and user\n  ``site-packages`` directory in ``sys.path``\n\n- The ``pydoc`` recipe has been beefed up to also exclude ``BaseHTTPServer``,\n  etc.\n\nKnown issues:\n\n- Commands marked with XXX in the help are not implemented\n\n- Includes *all* files from packages, it should be smart enough to strip\n  unused .py/.pyc/.pyo files (to save space, depending on which optimization\n  flag is used)\n\n- ``macholib`` should be refactored to use ``altgraph``\n\n- ``py2app.build_app`` and ``py2app.modulegraph`` should be refactored to\n  search for dependencies on a per-application basis\n\n.. _`graphlib`: http://www.personal.psu.edu/staff/i/u/iua1/python/graphlib/html/\n.. _`TinyTinyEdit example`: http://undefined.org/~bob/TinyTinyEdit.pdf\n\npy2app 0.1.2\n------------\n\n`py2app`_ 0.2 is primarily a bugfix release:\n\n- The encodings package now gets included in the zip file (saves space)\n\n- A copy of the Python interpreter is not included anymore in standalone\n  builds (saves space)\n\n- The executable bootstrap is now stripped by default (saves a little space)\n\n- ``sys.argv`` is set correctly now, it used to point to the executable, now\n  it points to the boot script.  This should enhance compatibility with some\n  applications.\n\n- Adds an "Alias" feature to modulegraph, so that ``sys.modules`` craziness\n  such as ``wxPython.wx -> wx`` can be accomodated (this particular craziness\n  is also now handled by default)\n\n- A ``sys.path`` alternative may be passed to ``find_modules`` now, though\n  this is not used yet\n\n- The ``Command`` instance is now passed to recipes instead of the\n  ``Distribution`` instance (though no recipes currently use either)\n\n- The post-filtering of modules and extensions is now generalized into a\n  stack and can be modified by recipes\n\n- A `wxPython`_ example demonstrating how to package `wxGlade`_ has been\n  added (this is a good example of how to write your own recipe, and how to\n  deal with complex applications that mix code and data files)\n\n- ``PyRuntimeLocations`` is now set to (only) the location of the current\n  interpreter\'s ``Python.framework`` for alias and semi-standalone build\n  modes (enhances compatibility with extensions built with an unpatched\n  Makefile with Mac OS X 10.3\'s Python 2.3.0)\n\nKnown issues:\n\n- Includes *all* files from packages, it should be smart enough to strip\n  unused .py/.pyc/.pyo files (to save space, depending on which optimization\n  flag is used).\n\n.. _`wxGlade`: http://wxglade.sourceforge.net/\n\npy2app 0.1.1\n------------\n\n`py2app`_ 0.1.1 is primarily a bugfix release:\n\n- Several problems related to Mac OS X 10.2 compatibility and standalone\n   building have been resolved\n\n- Scripts that are not in the same directory as setup.py now work\n\n- A new recipe has been added that removes the pydoc -> Tkinter dependency\n\n- A recipe has been added for `py2app`_ itself\n\n- a `wxPython`_ example (superdoodle) has been added.\n  Demonstrates not only how easy it is (finally!) to bundle\n  `wxPython`_ applications, but also how one setup.py can\n  deal with both `py2exe`_ and `py2app`_.\n\n- A new experimental tool, py2applet, has been added.\n  Once you\'ve built it (``python setup.py py2app``, of course), you should\n  be able to build simple applications simply by dragging your main script\n  and optionally any packages, data files, Info.plist and icon it needs.\n\nKnown issues:\n\n- Includes *all* files from packages, it should be smart enough to strip\n  unused .py/.pyc/.pyo files (to save space, depending on which\n  optimization flag is used).\n\n- The default ``PyRuntimeLocations`` can cause problems on machines that\n  have a /Library/Frameworks/Python.framework installed.  Workaround is\n  to set a plist that has the following key:\n  ``PyRuntimeLocations=[\'/System/Library/Frameworks/Python.framework/Versions/2.3/Python\']``\n  (this will be resolved soon)\n\n\npy2app 0.1\n----------\n\n(first public release)\n`py2app`_ is the bundlebuilder replacement we\'ve all been waiting\nfor.  It is implemented as a distutils command, similar to `py2exe`_.\n\n.. _`wxPython`: http://www.wxpython.org/\n.. _`py2app`: http://undefined.org/python/#py2app\n.. _`py2exe`: http://starship.python.net/crew/theller/py2exe/\n'), 'keywords': '.app,standalone', 'license': 'MIT or PSF License', 'name': 'py2app', 'url': 'http://github.com/ronaldoussoren/py2app', 'version': '0.22'}, 'sha1': '2a6712f5ddc72c54fb9f6aabab9cd5feb7e7eb13'}, 'doc_as_upsert': True}}}])

Anyway, I just tried flattened and it accepts documents, but doesn't alloww querying them the way we want, so I'm working on a different solution.

We tested with the latest search 0.3.3 but that failed [1]

It does not work, it seems expand expects simpler data structure [2]

Basic failure reproduction with ipython and an extract data sample
(simplified) out of the journal [3]

[1] P905

[2] P898

[3] Not exactly the same error as the swh search journal client run... but still

$ ipython
...
In [17]: data = {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0',
    ...:  'author': ({'email': 'cboettig@gmail.com',
    ...:              'familyName': 'Boettiger',
    ...:              'givenName': 'Carl',
    ...:              'id': 'http://orcid.org/0000-0002-1642-628X',
    ...:              'type': 'Person'},
    ...:             {'email': 'jones@nceas.ucsb.edu',
    ...:              'familyName': 'Jones',
    ...:              'givenName': 'Matthew B.',
    ...:              'id': 'http://orcid.org/0000-0003-0077-4738',
    ...:              'type': 'Person'})}

In [18]: from swh.indexer.codemeta import expand

In [19]: expand(data)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-369ca88a5be6> in <module>
----> 1 expand(data)

~/work/inria/repo/swh/swh-environment/swh-indexer/swh/indexer/codemeta.py in expand(doc)
    129 def expand(doc):
    130     """Same as `pyld.jsonld.expand`, but in the context of CodeMeta."""
--> 131     return jsonld.expand(doc, options={"documentLoader": _document_loader})
    132
    133

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in expand(input_, options)
    159     :return: the expanded JSON-LD output.
    160     """
--> 161     return JsonLdProcessor().expand(input_, options)
    162
    163

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in expand(self, input_, options)
    833
    834         # do expansion
--> 835         expanded = self._expand(active_ctx, None, document, options, False)
    836
    837         # optimize away @graph with no other properties

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in _expand(self, active_ctx, active_property, element, options, inside_list)
   2112         self._expand_object(
   2113             active_ctx, active_property, expanded_active_property,
-> 2114             element, rval, options, inside_list)
   2115
   2116         # get property count on expanded output

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in _expand_object(self, active_ctx, active_property, expanded_active_property, element, expanded_parent, options, inside_list)
   2423                     # recursively expand value w/key as new active property
   2424                     expanded_value = self._expand(
-> 2425                         term_ctx, key, value, options, inside_list=False)
   2426
   2427             # drop None values if property is not @value (dropped below)

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in _expand(self, active_ctx, active_property, element, options, inside_list)
   2082
   2083             # expand element according to value expansion rules
-> 2084             return self._expand_value(active_ctx, active_property, element)
   2085
   2086         # recursively expand object

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in _expand_value(self, active_ctx, active_property, value)
   2994
   2995         # do conversion of values that aren't basic JSON types to strings
-> 2996         if not (_is_bool(value) or _is_numeric(value) or _is_string(value)):
   2997             value = str(value)
   2998

~/.virtualenvs/swh/lib/python3.7/site-packages/pyld/jsonld.py in _is_numeric(v)
   5653     """
   5654     try:
-> 5655         float(v)
   5656         return True
   5657     except ValueError:

TypeError: float() argument must be a string or a number, not 'tuple'

why is there a tuple in the input? it should be a list

no idea, it's coming straight from the journal.
Note that it is not the only errors (P905 demonstrates another error).

There may be something fishy.

I suspect P905 might be the same issue. pyld tends to check if it's a list/array/string, and if it's not it assumes it's a number