Commit Graph

185 Commits

Author SHA1 Message Date
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Aman Agrawal 5ab62a3514 sgrep/semgrep: Updgrade and rename sgrep to semgrep.
With its new upgrade, sgrep has been renamed to semgrep.

Rename sgrep.yml to semgrep.yml
2020-05-01 11:01:14 -07:00
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Aman Agrawal 8944e0ad53 provision: Drop Xenial and stretch support. 2020-04-22 09:58:39 -07:00
arpit551 56a5beddc0 ci: Fixed errors related to running services in ci production build.
Restart postgres service if provision is called in production test suite.
This is required because terminate-psql-sessions script (used
in tools/ci/setup-production) throws error if postgres service is not running.

Restart rabbitmq service if provision is called in production test suite.
This is done to start the node as Circle CI don't start services on installation.

Removed memcached restart as flush-memcached script (which is furthur
used in tools/ci/production) throws UNKNOWN READ FAILURE if memcached is restarted
in development.
2020-04-21 14:46:40 -07:00
arpit551 dee8de3b88 ci: Replaced travis with test suite while setting production setup on ci. 2020-04-21 14:46:40 -07:00
Puneeth Chaganti 7a5033852d provision: Add dependencies for Chromium used by latest puppeteer.
a51902a870 upgraded the puppeteer version to
3.0.0, which upgraded the Chromium version being used[1]. This version of
Chromium requires a couple of extra libraries to be installed.

[1]: https://github.com/puppeteer/puppeteer/releases/tag/v3.0.0
2020-04-20 23:25:45 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
brendon f65e6d0d94 sgrep: Install syntactic code search tool as an external linter.
Add sgrep (sgrep.dev) to tooling and include simple rule as
proof of concept. Included rule detects use of old django render
function.

Also added a rule that looks for if-else statements where both
code paths are identical.
2020-04-01 15:08:34 -07:00
arpit551 a2dd1fad12 provision: switch Focal to use PGroonga from PPA.
Since Groonga packages for Ubuntu 20.04 Focal are now available
in their ppa so stopped building pgroonga from source.
2020-03-31 15:02:31 -07:00
Anders Kaseorg 43996106d8 requirements: Get transifex-client from apt.
In the past it has blocked Python library security updates with overly
strict version bounds, and we don’t use it as a library, only as a
binary.

Skip the PROVISION_VERSION bump because we can use the tx binary from
either location.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:39:04 -07:00
arpit551 b5c2202b3b provision: add dependency for building pgroonga from source.
clang-9 and llvm-9-dev are added for building pgroonga from source.
2020-03-25 16:09:59 -07:00
Tim Abbott 3d0b720543 provision: Fix missing apt update for installing new dependencies.
Previously, we only did apt updates when our sources.list files or
keys changed, which could result in provisioning errors for
development systems that don't routinely update their apt cache
(probably including ~all Vagrant environments).
2020-03-23 13:47:17 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
Dinesh 9e65f3f354 provision: Add provision support for puppeteer. 2020-03-22 11:42:14 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
arpit551 3963b6740a provision: Corrected how scripts/lib/build-* is included for sha1 sum check.
There were two problems with the previous code-
1) The code glob.glob("scripts/lib/build-") should be
   glob.glob("scripts/lib/build-*) otherwise it would always return [].
2) The part of the code where we included scripts/lib/build-* for sha1 sum
   check would only run when debian is not in os_families(). This wasn't
   correct as we could have a situation where we have to build pgroonga
   from source even in case of debian and so we need to improve the
   condition on it.
Now since we only have build-pgroonga there its better to just directly hash
its content with the condition of BUILD_PGROONGA_FROM_SOURCE.
2020-03-17 14:57:14 -07:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
rht 60563e9180 provision: Install postgresql-devel on CentOS as system dependency. 2020-01-07 13:25:25 -08:00
Tim Abbott 27b3c1a312 provision: Move install-shellcheck to proper directory.
Scripts in scripts/ should be exclusively code that used in
production, and this isn't.
2019-09-24 12:54:33 -07:00
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg e0d122217b provision: Use pre-built pgroonga on Debian 10.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:35:29 -07:00
Tim Abbott 8ef610f8c8 provision: Remove now-duplicate extra Ubuntu section. 2019-08-29 12:53:10 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Anders Kaseorg 349f22524e provision: Don’t rely on uid == gid assumption when invoking chown.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Anders Kaseorg 88793e80ea provision: Skip some rm, mkdir, chown commands if possible.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Anders Kaseorg 671844d667 provision: Add default-jre-headless for vnu-jar.
This was previously pulled in by yui-compressor.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-08 17:28:20 -07:00
Tim Abbott de0a41bc9c provision: Fix missing dependency on unzip.
Because this is often installed by default, we hadn't noticed that our
Slack importer doesn't run without it.

Thanks to Ray Kraesig for the report.
2019-08-08 10:49:20 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Wyatt Hoodes f80402c64d tools: Use mypy_extensions for NoReturn.
This is the right way to access it with support for older Python
versions where it might not be in typing yet.
2019-07-31 16:18:24 -07:00
Wyatt Hoodes 927feb615e provision: Comment why NoReturn can't import normally. 2019-07-25 17:48:20 -07:00
Anders Kaseorg 6d5a20ac62 requirements: Remove django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg c5a4b0501b provision: Start a new Python process after activating the virtualenv.
Mismatching imports from outside and inside the virtualenv in the same
process was causing segfaults after apparently benign changes to the
script!

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 12:16:55 -07:00
Anders Kaseorg ab89f40a66 generate-custom-icon-webfont: Replace with webpack webfonts-loader.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-18 12:00:00 -07:00
Amal Rajan 09049f1678 provision: Add provision support for Ubuntu 19.04
This commit adds support for Ubuntu 19.04 in the development
environment.
2019-07-13 16:44:09 -07:00
Wyatt Hoodes 886932fd23 provision: Update error output on failed `yarn install` reattempt. 2019-07-08 12:13:34 -07:00
Vishnu Ks 1f250a3fd2 casper: Add support for exporting results as XUnit XML file. 2019-07-07 22:30:01 -07:00
Wyatt Hoodes 40227171e6 provision: Improve error messaging when attempting to use Ubuntu Trusty.
As part of dropping support, we add appropriate error messaging when a
user attempts to provision while using trusty.  If the user is running
in Vagrant we append information on how to proceed.
2019-07-03 14:32:25 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Anders Kaseorg 7b2a605978 provision: Sort system dependencies deterministically.
set iteration order is randomized in Python ≥ 3.3.  That might or
might not have had the potential for causing rare probabilistic bugs,
but if nothing else, it made build logs harder to compare.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 15:04:44 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Anders Kaseorg caecd1c2ad install: Disable installation and provisioning on Ubuntu 14.04 Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 15:58:53 -07:00
Anders Kaseorg b4204e7cfd Remove legacy (pre-Vagrant) Docker development environment setup.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 10:58:02 -07:00
Wyatt Hoodes 6670b15234 provision: Clean up `var/` for the move to `var/<uuid>/test-backend`. 2019-06-24 12:42:00 -07:00
Vishnu Ks fdcae3b0b7 provision: Add cd /srv/zulip to .bashrc only if /srv/zulip exists. 2019-06-14 15:49:56 -07:00
Wyatt Hoodes 6b033c7909 test-backend: Add steps to deal with potential database leaks.
A function was written in `test_fixtures.py` to drop a test database
template if the corresponding database id doesn't belong to a file.
Alongside this fact, every file that is written is removed after 60
minutes.  Meaning any potential database template can never exist
longer than one hour.

This follow-up work was added to deal with the potential race
conditions when running `test-backend`.  Ensuring that all templates
are properly dealt with.

Essentially rewritten by tabbott for cleanliness.

Fixes the remainder of #12426.
2019-06-14 15:23:25 -07:00
Anders Kaseorg 407564086d provision: Move apt-get update from retry handler to setup-apt-repo.
This avoids unnecessarily alarming error messages if the apt cache is
missing.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:38 -07:00