experiments
/
STT
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

GitHub Actions for macOS

This commit is contained in:
Alexandre Lissy 2021-03-27 11:23:19 +01:00 committed by Reuben Morais
parent 654a83a294
commit cd80708546
39 changed files with 1374 additions and 58 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
.github/actions/build-tensorflow/action.yml vendored Normal file
View File

@ -0,0 +1,11 @@
name: "Build TensorFlow"
description: "Build TensorFlow Build"
inputs:
flavor:
description: "Build flavor"
required: true
runs:
using: "composite"
steps:
- run: ./taskcluster/tf_tc-build.sh ${{ inputs.flavor }}
shell: bash

37
.github/actions/check_artifact_exists/README.md vendored Normal file
View File

@ -0,0 +1,37 @@
Building and using a TensorFlow cache:
======================================
The present action will check the existence of an artifact in the list of the
repo artifacts. Since we don't want always to download the artifact, we can't
rely on the official download-artifact action.
Rationale:
----------
Because of the amount of code required to build TensorFlow, the library build
is split into two main parts to make it much faster to run PRs:
- a TensorFlow prebuild cache
- actual code of the library
The TensorFlow prebuild cache exists because building tensorflow (even just the
`libtensorflow_cpp.so`) is a huge amount of code and it will take several hours
even on decent systems. So we perform a cache build of it, because the
tensorflow version does not change that often.
However, each PR might have changes to the actual library code, so we rebuild
this everytime.
The `tensorflow_opt-macOS` job checks whether such build cache exists alrady.
Those cache are stored as artifacts because [GitHub Actions
cache](https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows)
has size limitations.
The `build-tensorflow-macOS` job has a dependency against the cache check to
know whether it needs to run an actual build or not.
Hacking:
--------
For hacking into the action, please follow the [GitHub JavaScript
Actions](https://docs.github.com/en/actions/creating-actions/creating-a-javascript-action#commit-tag-and-push-your-action-to-github)
and specifically the usage of `ncc`.

28
.github/actions/check_artifact_exists/action.yml vendored Normal file
View File

@ -0,0 +1,28 @@
name: "check/download artifacts"
description: "Check and download that an artifact exists"
inputs:
name:
description: "Artifact name"
required: true
github_token:
description: "GitHub token"
required: false
default: ${{ github.token }}
download:
description: "Should we download?"
required: false
default: false
path:
description: "Where to unpack the artifact"
required: false
default: "./"
repo:
description: "Repository name with owner (like actions/checkout)"
required: false
default: ${{ github.repository }}
outputs:
status:
description: "Status string of the artifact: 'missing' or 'found'"
runs:
using: "node12"
main: "dist/index.js"

BIN
.github/actions/check_artifact_exists/dist/index.js vendored Normal file
View File

Binary file not shown.

626
.github/actions/check_artifact_exists/dist/licenses.txt vendored Normal file
View File

@ -0,0 +1,626 @@
@actions/core
MIT
The MIT License (MIT)
Copyright 2019 GitHub
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@actions/github
MIT
@actions/http-client
MIT
Actions Http Client for Node.js
Copyright (c) GitHub, Inc.
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN
NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@octokit/auth-token
MIT
The MIT License
Copyright (c) 2019 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@octokit/core
MIT
The MIT License
Copyright (c) 2019 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@octokit/endpoint
MIT
The MIT License
Copyright (c) 2018 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@octokit/graphql
MIT
The MIT License
Copyright (c) 2018 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@octokit/plugin-paginate-rest
MIT
MIT License Copyright (c) 2019 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@octokit/plugin-rest-endpoint-methods
MIT
MIT License Copyright (c) 2019 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@octokit/request
MIT
The MIT License
Copyright (c) 2018 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@octokit/request-error
MIT
The MIT License
Copyright (c) 2019 Octokit contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@vercel/ncc
MIT
Copyright 2018 ZEIT, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
adm-zip
MIT
MIT License
Copyright (c) 2012 Another-D-Mention Software and other contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
before-after-hook
Apache-2.0
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2018 Gregor Martynus and other contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
deprecation
ISC
The ISC License
Copyright (c) Gregor Martynus and contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
filesize
BSD-3-Clause
Copyright (c) 2020, Jason Mulligan
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of filesize nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
is-plain-object
MIT
The MIT License (MIT)
Copyright (c) 2014-2017, Jon Schlinkert.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
node-fetch
MIT
The MIT License (MIT)
Copyright (c) 2016 David Frank
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
once
ISC
The ISC License
Copyright (c) Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
tunnel
MIT
The MIT License (MIT)
Copyright (c) 2012 Koichi Kobayashi
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
universal-user-agent
ISC
# [ISC License](https://spdx.org/licenses/ISC)
Copyright (c) 2018, Gregor Martynus (https://github.com/gr2m)
Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
wrappy
ISC
The ISC License
Copyright (c) Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

119
.github/actions/check_artifact_exists/main.js vendored Normal file
View File

@ -0,0 +1,119 @@
const core = require('@actions/core');
const github = require('@actions/github');
const AdmZip = require('adm-zip');
const filesize = require('filesize');
const pathname = require('path');
const fs = require('fs');
async function getGoodArtifacts(client, owner, repo, name) {
const goodWorkflowArtifacts = await client.paginate(
"GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts",
{
owner: owner,
repo: repo,
run_id: github.context.runId,
},
(workflowArtifacts) => {
// console.log(" ==> workflowArtifacts", workflowArtifacts);
return workflowArtifacts.data.filter((a) => {
// console.log("==> Artifact check", a);
return a.name == name
})
}
);
console.log("==> maybe goodWorkflowArtifacts:", goodWorkflowArtifacts);
if (goodWorkflowArtifacts.length > 0) {
return goodWorkflowArtifacts;
}
const goodRepoArtifacts = await client.paginate(
"GET /repos/{owner}/{repo}/actions/artifacts",
{
owner: owner,
repo: repo,
},
(repoArtifacts) => {
// console.log(" ==> repoArtifacts", repoArtifacts);
return repoArtifacts.data.filter((a) => {
// console.log("==> Artifact check", a);
return a.name == name
})
}
);
console.log("==> maybe goodRepoArtifacts:", goodRepoArtifacts);
if (goodRepoArtifacts.length > 0) {
return goodRepoArtifacts;
}
// We have not been able to find a repo artifact, it's really no good news
return [];
}
async function main() {
const token = core.getInput("github_token", { required: true });
const [owner, repo] = core.getInput("repo", { required: true }).split("/");
const path = core.getInput("path", { required: true });
const name = core.getInput("name");
const download = core.getInput("download");
const client = github.getOctokit(token)
console.log("==> Repo:", owner + "/" + repo);
const goodArtifacts = await getGoodArtifacts(client, owner, repo, name);
console.log("==> goodArtifacts:", goodArtifacts);
let artifactStatus = "";
if (goodArtifacts.length === 0) {
artifactStatus = "missing";
} else {
artifactStatus = "found";
}
console.log("==> Artifact", name, artifactStatus);
console.log("==> download", download);
core.setOutput("status", artifactStatus);
if (artifactStatus === "found" && download == "true") {
console.log("==> # artifacts:", goodArtifacts.length);
let artifact = goodArtifacts[0];
console.log("==> Artifact:", artifact.id)
const size = filesize(artifact.size_in_bytes, { base: 10 })
console.log("==> Downloading:", artifact.name + ".zip", `(${size})`)
const zip = await client.actions.downloadArtifact({
owner: owner,
repo: repo,
artifact_id: artifact.id,
archive_format: "zip",
})
const dir = name ? path : pathname.join(path, artifact.name)
fs.mkdirSync(dir, { recursive: true })
const adm = new AdmZip(Buffer.from(zip.data))
adm.getEntries().forEach((entry) => {
const action = entry.isDirectory ? "creating" : "inflating"
const filepath = pathname.join(dir, entry.entryName)
console.log(` ${action}: ${filepath}`)
})
adm.extractAllTo(dir, true)
}
if (artifactStatus === "missing" && download == "true") {
core.setFailed("Required", name, "that is missing");
}
return;
}
main();

11
.github/actions/check_artifact_exists/package.json vendored Normal file
View File

@ -0,0 +1,11 @@
{
"name": "check_artifact_exists",
"main": "main.js",
"devDependencies": {
"@actions/core": "^1.2.6",
"@actions/github": "^4.0.0",
"@vercel/ncc": "^0.27.0",
"adm-zip": "^0.5.2",
"filesize": "^6.1.0"
}
}

22
.github/actions/cpp-tests/action.yml vendored Normal file
View File

@ -0,0 +1,22 @@
name: "C++ tests"
description: "Running C++ binary tests"
inputs:
build-flavor:
description: "Running against TF or TFLite"
required: true
model-kind:
description: "Running against CI baked or production model"
required: true
bitrate:
description: "Bitrate for testing"
required: true
runs:
using: "composite"
steps:
- run: |
build=""
if [ "${{ inputs.build-flavor }}" = "tflite" ]; then
build="_tflite"
fi
./taskcluster/tc-cpp${build}-ds-tests${{ inputs.model-kind }}.sh ${{ inputs.bitrate }}
shell: bash

15
.github/actions/get_cache_key/README.md vendored Normal file
View File

@ -0,0 +1,15 @@
GitHub Action to compute cache key
==================================
It is intended to work in harmony with `check_artifact_exists`:
- compute a stable cache key
- as simple to use as possible (less parameters)
It will expect to be ran in a GitHub Action job that follows
`SUBMODULE_FLAVOR-PLATFORM`:
- it will use the `SUBMODULE` part to check what is the current SHA1 of this git submodule.
- the `FLAVOR` allows to distringuish e.g., opt/dbg builds
- the PLATFORM permits defining an os/arch couple
It allows for an `extras` field for extensive customization, like forcing a
re-build.

25
.github/actions/get_cache_key/action.yml vendored Normal file
View File

@ -0,0 +1,25 @@
name: "get cache key for submodule"
description: "Compute a cache key based on git submodule"
inputs:
extras:
description: "Extra cache key value"
required: true
outputs:
key:
description: "Computed cache key name"
value: ${{ steps.compute_cache_key.outputs.key }}
runs:
using: "composite"
steps:
- id: compute_cache_key
run: |
JOB=${{ github.job }}
SUBMODULE=$(echo $JOB | cut -d'-' -f1 | cut -d'_' -f1)
FLAVOR=$(echo $JOB | cut -d'-' -f1 | cut -d'_' -f2)
OSARCH=$(echo $JOB | cut -d'-' -f2)
SHA=$(git submodule status ${SUBMODULE} | sed -e 's/^-//g' -e 's/^+//g' -e 's/^U//g' | awk '{ print $1 }')
KEY=${SUBMODULE}-${FLAVOR}_${OSARCH}_${SHA}_${{ inputs.extras }}
echo "::set-output name=key::${KEY}"
shell: bash

11
.github/actions/host-build/action.yml vendored Normal file
View File

@ -0,0 +1,11 @@
name: "Host build lib"
description: "Host build of lib"
inputs:
flavor:
description: "Build flavor"
required: true
runs:
using: "composite"
steps:
- run: ./taskcluster/host-build.sh ${{ inputs.flavor }}
shell: bash

18
.github/actions/install-python-upstream/action.yml vendored Normal file
View File

@ -0,0 +1,18 @@
name: "Install Python"
description: "Installing an upstream python release"
inputs:
version:
description: "Python version"
required: true
runs:
using: "composite"
steps:
- shell: bash
run: curl https://www.python.org/ftp/python/${{ inputs.version }}/python-${{ inputs.version }}-macosx10.9.pkg -o "python.pkg"
- shell: bash
run: sudo installer -verbose -pkg python.pkg -target /
- shell: bash
run: |
which python3
python3 --version
python3 -c "import sysconfig; print(sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET'))"

60
.github/actions/node-build/action.yml vendored Normal file
View File

@ -0,0 +1,60 @@
name: "NodeJS binding"
description: "Binding a nodejs binding"
inputs:
nodejs_versions:
description: "NodeJS versions supported"
required: true
electronjs_versions:
description: "ElectronJS versions supported"
required: true
local_cflags:
description: "CFLAGS for NodeJS package"
required: false
default: ""
local_ldflags:
description: "LDFLAGS for NodeJS package"
required: false
default: ""
local_libs:
description: "LIBS for NodeJS package"
required: false
default: ""
runs:
using: "composite"
steps:
- run: |
node --version
npm --version
shell: bash
- run: |
npm update
shell: bash
- run: |
for node in ${{ inputs.nodejs_versions }}; do
EXTRA_CFLAGS=${{ inputs.local_cflags }} \
EXTRA_LDFLAGS=${{ inputs.local_ldflags }} \
EXTRA_LIBS=${{ inputs.local_libs }} \
make -C native_client/javascript \
NODE_ABI_TARGET=--target=${node} \
clean node-wrapper
done;
shell: bash
- run: |
for electron in ${{ inputs.electronjs_versions }}; do
EXTRA_CFLAGS=${{ inputs.local_cflags }} \
EXTRA_LDFLAGS=${{ inputs.local_ldflags }} \
EXTRA_LIBS=${{ inputs.local_libs }} \
make -C native_client/javascript \
NODE_ABI_TARGET=--target=${electron} \
NODE_DIST_URL=--disturl=https://electronjs.org/headers \
NODE_RUNTIME=--runtime=electron \
clean node-wrapper
done;
shell: bash
- run: |
make -C native_client/javascript clean npm-pack
shell: bash
- run: |
tar -czf native_client/javascript/wrapper.tar.gz \
-C native_client/javascript/ lib/
shell: bash

7
.github/actions/package-tensorflow/action.yml vendored Normal file
View File

@ -0,0 +1,7 @@
name: "Package TensorFlow"
description: "Package TensorFlow Build"
runs:
using: "composite"
steps:
- run: ./taskcluster/tf_tc-package.sh
shell: bash

7
.github/actions/package/action.yml vendored Normal file
View File

@ -0,0 +1,7 @@
name: "Package lib"
description: "Package of lib"
runs:
using: "composite"
steps:
- run: ./taskcluster/package.sh
shell: bash

47
.github/actions/python-build/action.yml vendored Normal file
View File

@ -0,0 +1,47 @@
name: "Python binding"
description: "Binding a python binding"
inputs:
build_flavor:
description: "Python package name"
required: true
local_cflags:
description: "CFLAGS for Python package"
required: false
default: ""
local_ldflags:
description: "LDFLAGS for Python package"
required: false
default: ""
local_libs:
description: "LIBS for Python package"
required: false
default: ""
runs:
using: "composite"
steps:
- run: |
python3 --version
pip3 --version
shell: bash
- run: |
mkdir -p wheels
shell: bash
- run: |
PROJECT_NAME="deepspeech"
if [ "${{ inputs.build_flavor }}" = "tflite" ]; then
PROJECT_NAME="deepspeech-tflite"
fi
EXTRA_CFLAGS=${{ inputs.local_cflags }} \
EXTRA_LDFLAGS=${{ inputs.local_ldflags }} \
EXTRA_LIBS=${{ inputs.local_libs }} \
make -C native_client/python/ \
SETUP_FLAGS="--project_name ${PROJECT_NAME}" \
bindings-clean bindings
shell: bash
- run: |
cp native_client/python/dist/*.whl wheels
shell: bash
- run: |
make -C native_client/python/ bindings-clean
shell: bash

11
.github/actions/select-xcode/action.yml vendored Normal file
View File

@ -0,0 +1,11 @@
name: "Select XCode version"
description: "Select XCode version"
inputs:
version:
description: "XCode version"
required: true
runs:
using: "composite"
steps:
- run: sudo xcode-select --switch /Applications/Xcode_${{ inputs.version }}.app
shell: bash

7
.github/actions/setup-tensorflow/action.yml vendored Normal file
View File

@ -0,0 +1,7 @@
name: "Setup TensorFlow"
description: "Setup TensorFlow Build"
runs:
using: "composite"
steps:
- run: ./taskcluster/tf_tc-setup.sh
shell: bash

245
.github/workflows/macOS-amd64.yml vendored Normal file
View File

@ -0,0 +1,245 @@
name: "macOS amd64"
on:
pull_request:
push:
branches:
- master
env:
TASKCLUSTER_TASK_DIR: ${{ github.workspace }}
TASKCLUSTER_ARTIFACTS: ${{ github.workspace }}/artifacts
MACOSX_DEPLOYMENT_TARGET: "10.10"
jobs:
swig_macOS:
name: "Build SWIG for macOS"
runs-on: macos-10.15
steps:
- uses: actions/checkout@v2
with:
repository: "swig/swig"
ref: "90cdbee6a69d13b39d734083b9f91069533b0d7b"
- run: |
brew install automake
- run: |
mkdir -p build-static/
- run: |
curl -sSL https://ftp.pcre.org/pub/pcre/pcre-8.43.tar.gz > pcre-8.43.tar.gz
- run: |
./Tools/pcre-build.sh
- run: |
sh autogen.sh
./configure \
--prefix=${{ github.workspace }}/build-static/ \
--program-prefix=ds-
- run: |
make -j
- run: |
make install
- uses: actions/upload-artifact@v2
with:
name: ${{ github.job }}
path: ${{ github.workspace }}/build-static/
tensorflow_opt-macOS:
name: "Check cache for TensorFlow"
runs-on: ubuntu-20.04
outputs:
status: ${{ steps.check_artifact_exists.outputs.status }}
cache_key: ${{ steps.get_cache_key.outputs.key }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- id: get_cache_key
uses: ./.github/actions/get_cache_key
with:
extras: "2"
- id: check_artifact_exists
uses: ./.github/actions/check_artifact_exists
with:
name: ${{ steps.get_cache_key.outputs.key }}
build-tensorflow-macOS:
name: "Build TensorFlow (opt) for macOS"
needs: tensorflow_opt-macOS
runs-on: macos-10.15
steps:
- run: true
if: needs.tensorflow_opt-macOS.outputs.status == 'found'
- uses: actions/checkout@v2
with:
fetch-depth: 0
submodules: 'recursive'
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
- uses: ./.github/actions/select-xcode
with:
version: "12.1.1"
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
- uses: ./.github/actions/setup-tensorflow
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
- uses: ./.github/actions/build-tensorflow
with:
flavor: "--darwin-cpu"
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
- uses: ./.github/actions/package-tensorflow
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
- uses: actions/upload-artifact@v2
with:
name: ${{ needs.tensorflow_opt-macOS.outputs.cache_key }}
path: ${{ github.workspace }}/artifacts/home.tar.xz
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
build-lib_macOS:
name: "Build libdeepspeech.so and deepspeech C++ binary"
runs-on: macos-10.15
needs: [ build-tensorflow-macOS, tensorflow_opt-macOS ]
strategy:
matrix:
build-flavor: ["tf", "tflite"]
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/download-artifact@v2
with:
name: ${{ needs.tensorflow_opt-macOS.outputs.cache_key }}
path: ${{ github.workspace }}/
if: needs.tensorflow_opt-macOS.outputs.status == 'missing'
- uses: ./.github/actions/check_artifact_exists
with:
name: ${{ needs.tensorflow_opt-macOS.outputs.cache_key }}
path: ${{ github.workspace }}/
download: true
if: needs.tensorflow_opt-macOS.outputs.status == 'found'
- run: |
tar xkf ${{ github.workspace }}/home.tar.xz
rm ${{ github.workspace }}/home.tar.xz
- run: |
git status
- uses: ./.github/actions/select-xcode
with:
version: "12.1.1"
- uses: ./.github/actions/host-build
with:
flavor: ${{ matrix.build-flavor }}
- uses: ./.github/actions/package
- uses: actions/upload-artifact@v2
with:
name: "native_client.${{ matrix.build-flavor }}.tar.xz"
path: ${{ github.workspace }}/artifacts/native_client.tar.xz
- uses: actions/upload-artifact@v2
with:
name: "libdeepspeech.${{ matrix.build-flavor }}.zip"
path: ${{ github.workspace }}/artifacts/libdeepspeech.zip
build-python-macOS:
name: "Build python bindings for macOS"
runs-on: macos-10.15
needs: [ build-lib_macOS, swig_macOS ]
strategy:
matrix:
build-flavor: ["tf", "tflite"]
python-version: [3.6.8, 3.7.9, 3.8.8, 3.9.2]
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/download-artifact@v2
with:
name: "native_client.${{ matrix.build-flavor }}.tar.xz"
path: ${{ github.workspace }}/tensorflow/bazel-bin/native_client/
- run: |
cd ${{ github.workspace }}/tensorflow/bazel-bin/native_client/
tar xf native_client.tar.xz
ls -hal
cd ${{ github.workspace }}/
- uses: actions/download-artifact@v2
with:
name: "swig_macOS"
path: ${{ github.workspace }}/native_client/ds-swig/
- run: |
ls -hal ${{ github.workspace }}/native_client/ds-swig/bin
ln -s ds-swig ${{ github.workspace }}/native_client/ds-swig/bin/swig
chmod +x ${{ github.workspace }}/native_client/ds-swig/bin/ds-swig ${{ github.workspace }}/native_client/ds-swig/bin/swig
- uses: ./.github/actions/install-python-upstream
with:
version: ${{ matrix.python-version }}
# GitHub packaged version are limited to macOS deployment target 10.14
#- uses: actions/setup-python@v2
# with:
# python-version: ${{ matrix.python-version }}
- uses: ./.github/actions/python-build
with:
build_flavor: ${{ matrix.build-flavor }}
- uses: actions/upload-artifact@v2
with:
name: "deepspeech-${{ matrix.build-flavor }}-${{ matrix.python-version }}.whl"
path: ${{ github.workspace }}/wheels/*.whl
build-nodejs-macOS:
name: "Build NodeJS and ElectronJS for macOS"
runs-on: macos-10.15
needs: [ build-lib_macOS, swig_macOS ]
strategy:
matrix:
build-flavor: ["tf", "tflite"]
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/download-artifact@v2
with:
name: "native_client.${{ matrix.build-flavor }}.tar.xz"
path: ${{ github.workspace }}/tensorflow/bazel-bin/native_client/
- run: |
cd ${{ github.workspace }}/tensorflow/bazel-bin/native_client/
tar xf native_client.tar.xz
ls -hal
cd ${{ github.workspace }}/
- uses: actions/download-artifact@v2
with:
name: "swig_macOS"
path: ${{ github.workspace }}/native_client/ds-swig/
- run: |
ls -hal ${{ github.workspace }}/native_client/ds-swig/bin
ln -s ds-swig ${{ github.workspace }}/native_client/ds-swig/bin/swig
chmod +x ${{ github.workspace }}/native_client/ds-swig/bin/ds-swig ${{ github.workspace }}/native_client/ds-swig/bin/swig
- uses: actions/setup-node@v2
with:
node-version: 12
- uses: ./.github/actions/node-build
with:
nodejs_versions: "10.0.0 11.0.0 12.7.0 13.0.0 14.0.0 15.0.0"
electronjs_versions: "5.0.13 6.0.12 6.1.7 7.0.1 7.1.8 8.0.1 9.0.1 9.1.0 9.2.0 10.0.0 10.1.0 11.0.0 12.0.0"
- uses: actions/upload-artifact@v2
with:
name: "nodewrapper-${{ matrix.build-flavor }}.tar.gz"
path: ${{ github.workspace }}/native_client/javascript/wrapper.tar.gz
- uses: actions/upload-artifact@v2
with:
name: "deepspeech-${{ matrix.build-flavor }}.tar.gz"
path: ${{ github.workspace }}/native_client/javascript/deepspeech-*.tgz
test-cpp-macOS:
name: "Test C++ binary on macOS"
runs-on: macos-10.15
needs: build-lib_macOS
strategy:
matrix:
build-flavor: ["tf", "tflite"]
models: ["-prod"]
bitrate: ["8k", "16k"]
env:
TASKCLUSTER_TMP_DIR: ${{ github.workspace }}/tmp/
DEEPSPEECH_PROD_MODEL: https://github.com/reuben/DeepSpeech/releases/download/v0.7.0-alpha.3/output_graph.pb
DEEPSPEECH_PROD_MODEL_MMAP: https://github.com/reuben/DeepSpeech/releases/download/v0.7.0-alpha.3/output_graph.pbmm
EXPECTED_TENSORFLOW_VERSION: "TensorFlow: v2.3.0-6-g23ad988"
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/download-artifact@v2
with:
name: "native_client.${{ matrix.build-flavor }}.tar.xz"
path: ${{ env.TASKCLUSTER_TMP_DIR }}
- run: |
cd ${{ env.TASKCLUSTER_TMP_DIR }}
mkdir ds && cd ds && tar xf ../native_client.tar.xz
- uses: ./.github/actions/cpp-tests
with:
build-flavor: ${{ matrix.build-flavor }}
bitrate: ${{ matrix.bitrate }}
model-kind: ${{ matrix.models }}

2
.gitignore vendored
View File

@ -39,3 +39,5 @@ doc/xml-java
doc/xml-dotnet doc/xml-dotnet
convert_graphdef_memmapped_format convert_graphdef_memmapped_format
native_client/swift/deepspeech_ios.framework/deepspeech_ios native_client/swift/deepspeech_ios.framework/deepspeech_ios
.github/actions/check_artifact_exists/node_modules/
.github/actions/check_artifact_exists/package-lock.json

12
native_client/ctcdecode/Makefile
View File

@ -44,16 +44,16 @@ workspace_status.cc:
# Enforce PATH here because swig calls from build_ext looses track of some # Enforce PATH here because swig calls from build_ext looses track of some
# variables over several runs # variables over several runs
bindings: clean-keep-third-party workspace_status.cc ds-swig bindings: clean-keep-third-party workspace_status.cc ds-swig
pip install --quiet $(PYTHON_PACKAGES) wheel==0.33.6 setuptools==45.0.0 pip3 install --quiet $(PYTHON_PACKAGES) wheel==0.33.6 setuptools==45.0.0
DISTUTILS_USE_SDK=1 PATH=$(DS_SWIG_BIN_PATH):$(TOOLCHAIN):$$PATH SWIG_LIB="$(SWIG_LIB)" AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python ./setup.py build_ext --num_processes $(NUM_PROCESSES) $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS) DISTUTILS_USE_SDK=1 PATH=$(DS_SWIG_BIN_PATH):$(TOOLCHAIN):$$PATH SWIG_LIB="$(SWIG_LIB)" AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python3 ./setup.py build_ext --num_processes $(NUM_PROCESSES) $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
find temp_build -type f -name "*.o" -delete find temp_build -type f -name "*.o" -delete
DISTUTILS_USE_SDK=1 AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python ./setup.py bdist_wheel $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS) DISTUTILS_USE_SDK=1 AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python3 ./setup.py bdist_wheel $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
rm -rf temp_build rm -rf temp_build
bindings-debug: clean-keep-third-party workspace_status.cc ds-swig bindings-debug: clean-keep-third-party workspace_status.cc ds-swig
pip install --quiet $(PYTHON_PACKAGES) wheel==0.33.6 setuptools==45.0.0 pip3 install --quiet $(PYTHON_PACKAGES) wheel==0.33.6 setuptools==45.0.0
DISTUTILS_USE_SDK=1 PATH=$(DS_SWIG_BIN_PATH):$(TOOLCHAIN):$$PATH SWIG_LIB="$(SWIG_LIB)" AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS) -DDEBUG" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python ./setup.py build_ext --debug --num_processes $(NUM_PROCESSES) $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS) DISTUTILS_USE_SDK=1 PATH=$(DS_SWIG_BIN_PATH):$(TOOLCHAIN):$$PATH SWIG_LIB="$(SWIG_LIB)" AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS) -DDEBUG" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python3 ./setup.py build_ext --debug --num_processes $(NUM_PROCESSES) $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
$(GENERATE_DEBUG_SYMS) $(GENERATE_DEBUG_SYMS)
find temp_build -type f -name "*.o" -delete find temp_build -type f -name "*.o" -delete
DISTUTILS_USE_SDK=1 AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS) -DDEBUG" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python ./setup.py bdist_wheel $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS) DISTUTILS_USE_SDK=1 AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) LIBEXE=$(LIBEXE) CFLAGS="$(CFLAGS) $(CXXFLAGS) -DDEBUG" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python3 ./setup.py bdist_wheel $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
rm -rf temp_build rm -rf temp_build

8
native_client/javascript/stt.i
View File

@ -46,7 +46,7 @@ using namespace node;
} }
%typemap(argout) ModelState **retval { %typemap(argout) ModelState **retval {
$result = SWIGV8_ARRAY_NEW(); $result = SWIGV8_ARRAY_NEW(0);
SWIGV8_AppendOutput($result, SWIG_From_int(result)); SWIGV8_AppendOutput($result, SWIG_From_int(result));
// owned by the application. NodeJS does not guarantee the finalizer will be called so applications must call FreeMetadata themselves. // owned by the application. NodeJS does not guarantee the finalizer will be called so applications must call FreeMetadata themselves.
%append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, 0)); %append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, 0));
@ -60,7 +60,7 @@ using namespace node;
} }
%typemap(argout) StreamingState **retval { %typemap(argout) StreamingState **retval {
$result = SWIGV8_ARRAY_NEW(); $result = SWIGV8_ARRAY_NEW(0);
SWIGV8_AppendOutput($result, SWIG_From_int(result)); SWIGV8_AppendOutput($result, SWIG_From_int(result));
// not owned, STT_FinishStream deallocates StreamingState // not owned, STT_FinishStream deallocates StreamingState
%append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, 0)); %append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, 0));
@ -70,14 +70,14 @@ using namespace node;
%nodefaultdtor ModelState; %nodefaultdtor ModelState;
%typemap(out) TokenMetadata* %{ %typemap(out) TokenMetadata* %{
$result = SWIGV8_ARRAY_NEW(); $result = SWIGV8_ARRAY_NEW(0);
for (int i = 0; i < arg1->num_tokens; ++i) { for (int i = 0; i < arg1->num_tokens; ++i) {
SWIGV8_AppendOutput($result, SWIG_NewPointerObj(SWIG_as_voidptr(&result[i]), SWIGTYPE_p_TokenMetadata, 0)); SWIGV8_AppendOutput($result, SWIG_NewPointerObj(SWIG_as_voidptr(&result[i]), SWIGTYPE_p_TokenMetadata, 0));
} }
%} %}
%typemap(out) CandidateTranscript* %{ %typemap(out) CandidateTranscript* %{
$result = SWIGV8_ARRAY_NEW(); $result = SWIGV8_ARRAY_NEW(0);
for (int i = 0; i < arg1->num_transcripts; ++i) { for (int i = 0; i < arg1->num_transcripts; ++i) {
SWIGV8_AppendOutput($result, SWIG_NewPointerObj(SWIG_as_voidptr(&result[i]), SWIGTYPE_p_CandidateTranscript, 0)); SWIGV8_AppendOutput($result, SWIG_NewPointerObj(SWIG_as_voidptr(&result[i]), SWIGTYPE_p_CandidateTranscript, 0));
} }

6
native_client/python/Makefile
View File

@ -9,8 +9,8 @@ bindings-clean:
# Enforce PATH here because swig calls from build_ext looses track of some # Enforce PATH here because swig calls from build_ext looses track of some
# variables over several runs # variables over several runs
bindings-build: ds-swig bindings-build: ds-swig
pip install --quiet $(PYTHON_PACKAGES) wheel==0.33.6 setuptools==45.0.0 pip3 install --quiet $(PYTHON_PACKAGES) wheel==0.33.6 setuptools==45.0.0
DISTUTILS_USE_SDK=1 PATH=$(TOOLCHAIN):$(DS_SWIG_BIN_PATH):$$PATH SWIG_LIB="$(SWIG_LIB)" AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED) $(RPATH_PYTHON)" MODEL_LDFLAGS="$(LDFLAGS_DIRS)" MODEL_LIBS="$(LIBS)" $(PYTHON_PATH) $(PYTHON_SYSCONFIGDATA) $(NUMPY_INCLUDE) python ./setup.py build_ext $(PYTHON_PLATFORM_NAME) DISTUTILS_USE_SDK=1 PATH=$(TOOLCHAIN):$(DS_SWIG_BIN_PATH):$$PATH SWIG_LIB="$(SWIG_LIB)" AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED) $(RPATH_PYTHON)" MODEL_LDFLAGS="$(LDFLAGS_DIRS)" MODEL_LIBS="$(LIBS)" $(PYTHON_PATH) $(PYTHON_SYSCONFIGDATA) $(NUMPY_INCLUDE) python3 ./setup.py build_ext $(PYTHON_PLATFORM_NAME)
MANIFEST.in: bindings-build MANIFEST.in: bindings-build
> $@ > $@
@ -21,6 +21,6 @@ MANIFEST.in: bindings-build
bindings-package: MANIFEST.in bindings-package: MANIFEST.in
cat MANIFEST.in cat MANIFEST.in
rm -f temp_build/*_wrap.o temp_build/Release/*_wrap.obj rm -f temp_build/*_wrap.o temp_build/Release/*_wrap.obj
DISTUTILS_USE_SDK=1 AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED) $(RPATH_PYTHON)" MODEL_LDFLAGS="$(LDFLAGS_DIRS)" MODEL_LIBS="$(LIBS)" $(PYTHON_PATH) $(PYTHON_SYSCONFIGDATA) $(NUMPY_INCLUDE) python ./setup.py bdist_wheel $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS) DISTUTILS_USE_SDK=1 AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED) $(RPATH_PYTHON)" MODEL_LDFLAGS="$(LDFLAGS_DIRS)" MODEL_LIBS="$(LIBS)" $(PYTHON_PATH) $(PYTHON_SYSCONFIGDATA) $(NUMPY_INCLUDE) python3 ./setup.py bdist_wheel $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
bindings: bindings-build bindings-package bindings: bindings-build bindings-package

15
taskcluster/host-build.sh
View File

@ -25,10 +25,11 @@ do_bazel_build
do_deepspeech_binary_build do_deepspeech_binary_build
if [ "${runtime}" = "tflite" ]; then # TODO: Disabled for now
do_deepspeech_python_build "--tflite" #if [ "${runtime}" = "tflite" ]; then
else # do_deepspeech_python_build "--tflite"
do_deepspeech_python_build #else
fi # do_deepspeech_python_build
#fi
do_deepspeech_nodejs_build #
#do_deepspeech_nodejs_build

12
taskcluster/package.sh
View File

@ -6,17 +6,17 @@ source $(dirname "$0")/tc-tests-utils.sh
mkdir -p ${TASKCLUSTER_ARTIFACTS} || true mkdir -p ${TASKCLUSTER_ARTIFACTS} || true
cp ${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel*.log ${TASKCLUSTER_ARTIFACTS}/ cp ${DS_DSDIR}/tensorflow/bazel*.log ${TASKCLUSTER_ARTIFACTS}/
package_native_client "native_client.tar.xz" package_native_client "native_client.tar.xz"
package_libdeepspeech_as_zip "libdeepspeech.zip" package_libdeepspeech_as_zip "libdeepspeech.zip"
if [ -d ${DS_ROOT_TASK}/DeepSpeech/ds/wheels ]; then if [ -d ${DS_DSDIR}/wheels ]; then
cp ${DS_ROOT_TASK}/DeepSpeech/ds/wheels/* ${TASKCLUSTER_ARTIFACTS}/ cp ${DS_DSDIR}/wheels/* ${TASKCLUSTER_ARTIFACTS}/
cp ${DS_ROOT_TASK}/DeepSpeech/ds/native_client/javascript/deepspeech-*.tgz ${TASKCLUSTER_ARTIFACTS}/ cp ${DS_DSDIR}/native_client/javascript/deepspeech-*.tgz ${TASKCLUSTER_ARTIFACTS}/
fi; fi;
if [ -f ${DS_ROOT_TASK}/DeepSpeech/ds/native_client/javascript/wrapper.tar.gz ]; then if [ -f ${DS_DSDIR}/native_client/javascript/wrapper.tar.gz ]; then
cp ${DS_ROOT_TASK}/DeepSpeech/ds/native_client/javascript/wrapper.tar.gz ${TASKCLUSTER_ARTIFACTS}/ cp ${DS_DSDIR}/native_client/javascript/wrapper.tar.gz ${TASKCLUSTER_ARTIFACTS}/
fi; fi;

22
taskcluster/tc-all-utils.sh
View File

@ -100,18 +100,18 @@ download_data()
local _model_source_mmap_file=$(basename "${model_source_mmap}") local _model_source_mmap_file=$(basename "${model_source_mmap}")
${WGET} "${model_source_mmap}" -O - | gunzip --force > "${TASKCLUSTER_TMP_DIR}/${_model_source_mmap_file}" ${WGET} "${model_source_mmap}" -O - | gunzip --force > "${TASKCLUSTER_TMP_DIR}/${_model_source_mmap_file}"
cp ${DS_ROOT_TASK}/DeepSpeech/ds/data/smoke_test/*.wav ${TASKCLUSTER_TMP_DIR}/ cp ${DS_DSDIR}/data/smoke_test/*.wav ${TASKCLUSTER_TMP_DIR}/
cp ${DS_ROOT_TASK}/DeepSpeech/ds/data/smoke_test/pruned_lm.scorer ${TASKCLUSTER_TMP_DIR}/kenlm.scorer cp ${DS_DSDIR}/data/smoke_test/pruned_lm.scorer ${TASKCLUSTER_TMP_DIR}/kenlm.scorer
cp ${DS_ROOT_TASK}/DeepSpeech/ds/data/smoke_test/pruned_lm.bytes.scorer ${TASKCLUSTER_TMP_DIR}/kenlm.bytes.scorer cp ${DS_DSDIR}/data/smoke_test/pruned_lm.bytes.scorer ${TASKCLUSTER_TMP_DIR}/kenlm.bytes.scorer
cp -R ${DS_ROOT_TASK}/DeepSpeech/ds/native_client/test ${TASKCLUSTER_TMP_DIR}/test_sources cp -R ${DS_DSDIR}/native_client/test ${TASKCLUSTER_TMP_DIR}/test_sources
} }
download_material() download_material()
{ {
target_dir=$1 target_dir=$1
download_native_client_files "${target_dir}" # TODO: FIXME download_native_client_files "${target_dir}"
download_data download_data
ls -hal ${TASKCLUSTER_TMP_DIR}/${model_name} ${TASKCLUSTER_TMP_DIR}/${model_name_mmap} ${TASKCLUSTER_TMP_DIR}/LDC93S1*.wav ls -hal ${TASKCLUSTER_TMP_DIR}/${model_name} ${TASKCLUSTER_TMP_DIR}/${model_name_mmap} ${TASKCLUSTER_TMP_DIR}/LDC93S1*.wav
@ -154,22 +154,22 @@ verify_bazel_rebuild()
mkdir -p ${TASKCLUSTER_ARTIFACTS} || true mkdir -p ${TASKCLUSTER_ARTIFACTS} || true
cp ${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel*.log ${TASKCLUSTER_ARTIFACTS}/ cp ${DS_DSDIR}/tensorflow/bazel*.log ${TASKCLUSTER_ARTIFACTS}/
spurious_rebuilds=$(grep 'Executing action' "${bazel_explain_file}" | grep 'Compiling' | grep -v -E 'no entry in the cache|[for host]|unconditional execution is requested|Executing genrule //native_client:workspace_status|Compiling native_client/workspace_status.cc|Linking native_client/libstt.so' | wc -l) spurious_rebuilds=$(grep 'Executing action' "${bazel_explain_file}" | grep 'Compiling' | grep -v -E 'no entry in the cache|[for host]|unconditional execution is requested|Executing genrule //native_client:workspace_status|Compiling native_client/workspace_status.cc|Linking native_client/libstt.so' | wc -l)
if [ "${spurious_rebuilds}" -ne 0 ]; then if [ "${spurious_rebuilds}" -ne 0 ]; then
echo "Bazel rebuilds some file it should not, please check." echo "Bazel rebuilds some file it should not, please check."
if is_patched_bazel; then if is_patched_bazel; then
mkdir -p ${DS_ROOT_TASK}/DeepSpeech/ckd/ds ${DS_ROOT_TASK}/DeepSpeech/ckd/tf mkdir -p ${DS_ROOT_TASK}/ckd/ds ${DS_ROOT_TASK}/ckd/tf
tar xf ${DS_ROOT_TASK}/DeepSpeech/bazel-ckd-tf.tar --strip-components=4 -C ${DS_ROOT_TASK}/DeepSpeech/ckd/ds/ tar xf ${DS_ROOT_TASK}/bazel-ckd-tf.tar --strip-components=4 -C ${DS_ROOT_TASK}/ckd/ds/
tar xf ${DS_ROOT_TASK}/DeepSpeech/bazel-ckd-ds.tar --strip-components=4 -C ${DS_ROOT_TASK}/DeepSpeech/ds/ckd/tensorflow/ tar xf ${DS_ROOT_TASK}/bazel-ckd-ds.tar --strip-components=4 -C ${DS_DSDIR}/ckd/tensorflow/
echo "Making a diff between CKD files" echo "Making a diff between CKD files"
mkdir -p ${TASKCLUSTER_ARTIFACTS} mkdir -p ${TASKCLUSTER_ARTIFACTS}
diff -urNw ${DS_ROOT_TASK}/DeepSpeech/ds/ckd/tensorflow/ ${DS_ROOT_TASK}/DeepSpeech/ckd/ds/ | tee ${TASKCLUSTER_ARTIFACTS}/ckd.diff diff -urNw ${DS_DSDIR}/ckd/tensorflow/ ${DS_ROOT_TASK}/ckd/ds/ | tee ${TASKCLUSTER_ARTIFACTS}/ckd.diff
rm -fr ${DS_ROOT_TASK}/DeepSpeech/ds/ckd/tensorflow/ ${DS_ROOT_TASK}/DeepSpeech/ckd/ds/ rm -fr ${DS_DSDIR}/ckd/tensorflow/ ${DS_ROOT_TASK}/ckd/ds/
else else
echo "Cannot get CKD information from release, please use patched Bazel" echo "Cannot get CKD information from release, please use patched Bazel"
fi; fi;

15
taskcluster/tc-all-vars.sh
View File

@ -49,9 +49,9 @@ export ANDROID_TMP_DIR=/data/local/tmp
mkdir -p ${TASKCLUSTER_TMP_DIR} || true mkdir -p ${TASKCLUSTER_TMP_DIR} || true
export DS_TFDIR=${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow export DS_TFDIR=${DS_ROOT_TASK}/tensorflow
export DS_DSDIR=${DS_ROOT_TASK}/DeepSpeech/ds export DS_DSDIR=${DS_ROOT_TASK}/
export DS_EXAMPLEDIR=${DS_ROOT_TASK}/DeepSpeech/examples export DS_EXAMPLEDIR=${DS_ROOT_TASK}/examples
export DS_VERSION="$(cat ${DS_DSDIR}/training/coqui_stt_training/VERSION)" export DS_VERSION="$(cat ${DS_DSDIR}/training/coqui_stt_training/VERSION)"
@ -61,10 +61,15 @@ export ANDROID_NDK_HOME=${DS_ROOT_TASK}/DeepSpeech/Android/android-ndk-r18b/
WGET=${WGET:-"wget"} WGET=${WGET:-"wget"}
TAR=${TAR:-"tar"} TAR=${TAR:-"tar"}
XZ=${XZ:-"pixz -9"} XZ=${XZ:-"xz -9 -T0"}
UNXZ=${UNXZ:-"pixz -d"} ZIP=${ZIP:-"zip"}
UNXZ=${UNXZ:-"xz -T0 -d"}
UNGZ=${UNGZ:-"gunzip"} UNGZ=${UNGZ:-"gunzip"}
if [ "${OS}" = "Darwin" ]; then
TAR="gtar"
fi
if [ "${OS}" = "${TC_MSYS_VERSION}" ]; then if [ "${OS}" = "${TC_MSYS_VERSION}" ]; then
WGET=/usr/bin/wget.exe WGET=/usr/bin/wget.exe
TAR=/usr/bin/tar.exe TAR=/usr/bin/tar.exe

2
taskcluster/tc-asserts.sh
View File

@ -227,7 +227,7 @@ assert_correct_ldc93s1_prodtflitemodel()
fi; fi;
if [ "$3" = "8k" ]; then if [ "$3" = "8k" ]; then
assert_correct_inference "$1" "she had to do so in greasy wash water all year" "$2" assert_correct_inference "$1" "she had to do so and greasy wash water all year" "$2"
fi; fi;
} }

6
taskcluster/tc-build-utils.sh
View File

@ -188,7 +188,7 @@ do_bazel_build()
if [ "${_opt_or_dbg}" = "opt" ]; then if [ "${_opt_or_dbg}" = "opt" ]; then
if is_patched_bazel; then if is_patched_bazel; then
find ${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel-out/ -iname "*.ckd" | tar -cf ${DS_ROOT_TASK}/DeepSpeech/bazel-ckd-tf.tar -T - find ${DS_ROOT_TASK}/tensorflow/bazel-out/ -iname "*.ckd" | tar -cf ${DS_ROOT_TASK}/bazel-ckd-tf.tar -T -
fi; fi;
fi; fi;
@ -197,9 +197,9 @@ do_bazel_build()
if [ "${_opt_or_dbg}" = "opt" ]; then if [ "${_opt_or_dbg}" = "opt" ]; then
if is_patched_bazel; then if is_patched_bazel; then
find ${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel-out/ -iname "*.ckd" | tar -cf ${DS_ROOT_TASK}/DeepSpeech/bazel-ckd-ds.tar -T - find ${DS_ROOT_TASK}/tensorflow/bazel-out/ -iname "*.ckd" | tar -cf ${DS_ROOT_TASK}/bazel-ckd-ds.tar -T -
fi; fi;
verify_bazel_rebuild "${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel_monolithic.log" verify_bazel_rebuild "${DS_ROOT_TASK}/tensorflow/bazel_monolithic.log"
fi; fi;
} }

0
taskcluster/tc-cpp-ds-tests-prod.sh Normal file → Executable file
View File

0
taskcluster/tc-cpp-ds-tests.sh Normal file → Executable file
View File

0
taskcluster/tc-cpp_tflite-tests-prod.sh → taskcluster/tc-cpp_tflite-ds-tests-prod.sh Normal file → Executable file
View File

0
taskcluster/tc-cpp_tflite-ds-tests.sh Normal file → Executable file
View File

0
taskcluster/tc-cpp_tflite_basic-ds-tests.sh Normal file → Executable file
View File

13
taskcluster/tc-package.sh
View File

@ -21,9 +21,14 @@ package_native_client()
echo "Please specify artifact name." echo "Please specify artifact name."
fi; fi;
win_lib=""
if [ -f "${tensorflow_dir}/bazel-bin/native_client/libstt.so.if.lib" ]; then
win_lib="-C ${tensorflow_dir}/bazel-bin/native_client/ libstt.so.if.lib"
fi;
${TAR} -cf - \ ${TAR} -cf - \
-C ${tensorflow_dir}/bazel-bin/native_client/ libstt.so \ -C ${tensorflow_dir}/bazel-bin/native_client/ libstt.so \
-C ${tensorflow_dir}/bazel-bin/native_client/ libstt.so.if.lib \ ${win_lib} \
-C ${tensorflow_dir}/bazel-bin/native_client/ generate_scorer_package \ -C ${tensorflow_dir}/bazel-bin/native_client/ generate_scorer_package \
-C ${deepspeech_dir}/ LICENSE \ -C ${deepspeech_dir}/ LICENSE \
-C ${deepspeech_dir}/native_client/ stt${PLATFORM_EXE_SUFFIX} \ -C ${deepspeech_dir}/native_client/ stt${PLATFORM_EXE_SUFFIX} \
@ -55,7 +60,7 @@ package_native_client_ndk()
echo "Please specify arch abi." echo "Please specify arch abi."
fi; fi;
tar -cf - \ ${TAR} -cf - \
-C ${deepspeech_dir}/native_client/libs/${arch_abi}/ stt \ -C ${deepspeech_dir}/native_client/libs/${arch_abi}/ stt \
-C ${deepspeech_dir}/native_client/libs/${arch_abi}/ libstt.so \ -C ${deepspeech_dir}/native_client/libs/${arch_abi}/ libstt.so \
-C ${tensorflow_dir}/bazel-bin/native_client/ generate_scorer_package \ -C ${tensorflow_dir}/bazel-bin/native_client/ generate_scorer_package \
@ -63,7 +68,7 @@ package_native_client_ndk()
-C ${deepspeech_dir}/native_client/ coqui-stt.h \ -C ${deepspeech_dir}/native_client/ coqui-stt.h \
-C ${deepspeech_dir}/ LICENSE \ -C ${deepspeech_dir}/ LICENSE \
-C ${deepspeech_dir}/native_client/kenlm/ README.coqui \ -C ${deepspeech_dir}/native_client/kenlm/ README.coqui \
| pixz -9 > "${artifacts_dir}/${artifact_name}" | ${XZ} > "${artifacts_dir}/${artifact_name}"
} }
package_libdeepspeech_as_zip() package_libdeepspeech_as_zip()
@ -83,5 +88,5 @@ package_libdeepspeech_as_zip()
echo "Please specify artifact name." echo "Please specify artifact name."
fi; fi;
zip -r9 --junk-paths "${artifacts_dir}/${artifact_name}" ${tensorflow_dir}/bazel-bin/native_client/libstt.so ${ZIP} -r9 --junk-paths "${artifacts_dir}/${artifact_name}" ${tensorflow_dir}/bazel-bin/native_client/libstt.so
} }

2
taskcluster/tf_tc-build.sh
View File

@ -4,7 +4,7 @@ set -ex
source $(dirname $0)/tf_tc-vars.sh source $(dirname $0)/tf_tc-vars.sh
pushd ${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/ pushd ${DS_ROOT_TASK}/tensorflow/
BAZEL_BUILD="bazel ${BAZEL_OUTPUT_USER_ROOT} build -s --explain bazel_monolithic_tf.log --verbose_explanations --experimental_strict_action_env --config=monolithic" BAZEL_BUILD="bazel ${BAZEL_OUTPUT_USER_ROOT} build -s --explain bazel_monolithic_tf.log --verbose_explanations --experimental_strict_action_env --config=monolithic"
# Start a bazel process to ensure reliability on Windows and avoid: # Start a bazel process to ensure reliability on Windows and avoid:

4
taskcluster/tf_tc-package.sh
View File

@ -6,9 +6,9 @@ source $(dirname $0)/tf_tc-vars.sh
mkdir -p ${TASKCLUSTER_ARTIFACTS} || true mkdir -p ${TASKCLUSTER_ARTIFACTS} || true
cp ${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel_*.log ${TASKCLUSTER_ARTIFACTS} cp ${DS_ROOT_TASK}/tensorflow/bazel_*.log ${TASKCLUSTER_ARTIFACTS} || true
OUTPUT_ROOT="${DS_ROOT_TASK}/DeepSpeech/ds/tensorflow/bazel-bin" OUTPUT_ROOT="${DS_ROOT_TASK}/tensorflow/bazel-bin"
for output_bin in \ for output_bin in \
tensorflow/lite/experimental/c/libtensorflowlite_c.so \ tensorflow/lite/experimental/c/libtensorflowlite_c.so \

4
taskcluster/tf_tc-setup.sh
View File

@ -106,10 +106,6 @@ fi
mkdir -p ${TASKCLUSTER_ARTIFACTS} || true mkdir -p ${TASKCLUSTER_ARTIFACTS} || true
pushd ${DS_ROOT_TASK}/DeepSpeech/ds/
git submodule --quiet sync tensorflow/ && git submodule --quiet update --init tensorflow/
popd
# Taken from https://www.tensorflow.org/install/source # Taken from https://www.tensorflow.org/install/source
# Only future is needed for our builds, as we don't build the Python package # Only future is needed for our builds, as we don't build the Python package

2
taskcluster/tf_tc-vars.sh
View File

@ -74,7 +74,7 @@ elif [ "${OS}" = "Darwin" ]; then
SHA_SUM="shasum -a 256 -c" SHA_SUM="shasum -a 256 -c"
WGET=wget WGET=wget
TAR=gtar TAR=gtar
XZ="pixz -9" XZ="xz -9 -T0"
fi; fi;
# /tmp/artifacts for docker-worker on linux, # /tmp/artifacts for docker-worker on linux,