Commit Graph

382 Commits

Author SHA1 Message Date
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job.
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
2020-04-08 12:52:31 -07:00
Vishnu KS 8415a1472a tools: Rename update-authors-json to fetch-contributor-data. 2020-04-08 12:40:00 -07:00
Greg Price daf357aaab tools: Give `require_clean_work_tree` our nice UI touches.
Notably, print the details of what's modified.
2020-04-06 17:52:10 -07:00
Greg Price 43ca39c87d tools: Modernize require_clean_work_tree a bit.
Use `local`.  Also `set -u`-compatible `${2-}`, and normalize
formatting of `if .. then`.
2020-04-06 17:52:10 -07:00
Greg Price 73acca76ba tools: Add a require_clean_work_tree function, from Git upstream.
This is verbatim from Git upstream, at an older version.  (The one
change since then is to add localization for the messages like "You
have unstaged changes" -- which complicates the code, is important and
worth it for Git itself, but for our tools we can do without.)

This function will replace our use of `git diff-index --quiet HEAD`
in several scripts.  The key differences in behavior are:

 * The `git update-index --refresh`.  Without this, on Windows
   apparently `git diff-index` routinely (but not all the time!)
   reports that tons of files have changed.  See report:
     https://chat.zulip.org/#narrow/stream/9-issues/topic/.2E.2Ftools.2Ffetch-pull-request.20issue/near/834435

 * Instead of one command comparing the worktree to HEAD, we
   separately compare the worktree to the index and the index to
   HEAD, and abort if either diff is nonempty.  This one is obvious,
   but rather an edge case (it matters only if you've managed to
   make the worktree and HEAD agree while the index has some
   changes), and the extra code is annoying if written out in every
   script that needs it.  But that's what a subroutine is for. :-)

We'll make a few tweaks before actually switching to use this.
2020-04-06 17:52:10 -07:00
brendon f65e6d0d94 sgrep: Install syntactic code search tool as an external linter.
Add sgrep (sgrep.dev) to tooling and include simple rule as
proof of concept. Included rule detects use of old django render
function.

Also added a rule that looks for if-else statements where both
code paths are identical.
2020-04-01 15:08:34 -07:00
arpit551 a2dd1fad12 provision: switch Focal to use PGroonga from PPA.
Since Groonga packages for Ubuntu 20.04 Focal are now available
in their ppa so stopped building pgroonga from source.
2020-03-31 15:02:31 -07:00
Steve Howell 6b86d9055f tests: Only wait up to 90s for the server. 2020-03-29 12:17:16 -07:00
Steve Howell cef74e1fdd tests: Sleep a bit more between server-is-up checks.
A watched pot never boils.

If we get server startup time down enough where 400ms
becomes a significant chunk, we can re-tweak this.
2020-03-29 12:17:16 -07:00
Steve Howell 7e4d542df4 tests: Use zulipdev.com for server_is_up() check.
Folks can have issues connecting to Casper
as zulipdev.com when they are not connected to
the internet or just have a bad connection, since
the DNS record is on the internet.  Folks can
work around this by just creating an /etc/hosts
entry for zulipdev.com, but people don't always
know.

This fix moves the symptom slightly earlier in
the process--we don't advertise that the server
is "up" if you can't actually connect to it as
"zulipdev.com".
2020-03-29 12:17:16 -07:00
Anders Kaseorg 43996106d8 requirements: Get transifex-client from apt.
In the past it has blocked Python library security updates with overly
strict version bounds, and we don’t use it as a library, only as a
binary.

Skip the PROVISION_VERSION bump because we can use the tx binary from
either location.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:39:04 -07:00
arpit551 b5c2202b3b provision: add dependency for building pgroonga from source.
clang-9 and llvm-9-dev are added for building pgroonga from source.
2020-03-25 16:09:59 -07:00
Tim Abbott 3d0b720543 provision: Fix missing apt update for installing new dependencies.
Previously, we only did apt updates when our sources.list files or
keys changed, which could result in provisioning errors for
development systems that don't routinely update their apt cache
(probably including ~all Vagrant environments).
2020-03-23 13:47:17 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
Dinesh 9e65f3f354 provision: Add provision support for puppeteer. 2020-03-22 11:42:14 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
arpit551 3963b6740a provision: Corrected how scripts/lib/build-* is included for sha1 sum check.
There were two problems with the previous code-
1) The code glob.glob("scripts/lib/build-") should be
   glob.glob("scripts/lib/build-*) otherwise it would always return [].
2) The part of the code where we included scripts/lib/build-* for sha1 sum
   check would only run when debian is not in os_families(). This wasn't
   correct as we could have a situation where we have to build pgroonga
   from source even in case of debian and so we need to improve the
   condition on it.
Now since we only have build-pgroonga there its better to just directly hash
its content with the condition of BUILD_PGROONGA_FROM_SOURCE.
2020-03-17 14:57:14 -07:00
Tim Abbott aaee506fb2 provision: Always build pygments_data.json if missing.
While it's a bit of extra complexity to do this check, which I'm not
excited about, we've had multiple folks spend significant time being
confused rebasing past d7d8632525 into
deleting `pygments_data.json`, with provision not rebuilding it, so
this seems worth merging as a transitional fix even if we decide to
remove it in 2 months.
2020-02-20 10:25:11 -08:00
Hashir Sarwar eb23c6fa6c test_fixtures: Clean up interface for `template_database_status()`.
1) Created a new class `DatabaseType` and access its objects inside
`template_database_status()` instead of sending five arguments with
default values.

2) Made `check_files` and `setting_name` local variables instead of
function parameters since they had same value(None) for every call.

Fixes #13845.
2020-02-12 11:07:10 -08:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
Aditya Bansal d0492b103b template_parser: Fix Jinja2 tag matching with strip whitespace syntax.
In this commit, we basically match any kinda of jinja2 start tag,
    no matter its special kind (eg. jinja2_whitespace_stripped_start)
    to any kinda jinja2 end tag (eg. jinja2_whitespace_stripped_end)

    Idea is special operators like `-` do not change the meaning of
    inline tag and thus matching shouldn't depend upon this.
2020-01-16 12:53:34 -08:00
Steve Howell 752d6dc6df tools: Remove find-add-class tool.
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.

At this point the tool is just a little too noisy
to be helpful.  An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS.  I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.

I just don't think it's worth maintaining the tool
at this point.
2020-01-14 15:45:49 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
rht 60563e9180 provision: Install postgresql-devel on CentOS as system dependency. 2020-01-07 13:25:25 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Anders Kaseorg 40f4ead738 mypy: Upgrade from 0.720 to 0.730.
Fixes #13269.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Ray Kraesig 1180299fe4 provisioning: fix up bash profile creation
The base Ubuntu image includes no `.bash_profile`. Creating one will
prevent the existing support scripts in `~/.profile` from loading.

Instead, append our setup code to [whichever script Bash will load
first][bash-startup].

[bash-startup]: https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html#Bash-Startup-Files
2019-10-08 17:17:55 -07:00
Tim Abbott 27b3c1a312 provision: Move install-shellcheck to proper directory.
Scripts in scripts/ should be exclusively code that used in
production, and this isn't.
2019-09-24 12:54:33 -07:00
Anders Kaseorg 9b38bcc2a6 templates: Abstract render_bundle behind a Jinja2 macro.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg e0d122217b provision: Use pre-built pgroonga on Debian 10.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:35:29 -07:00
Tim Abbott 8ef610f8c8 provision: Remove now-duplicate extra Ubuntu section. 2019-08-29 12:53:10 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Anders Kaseorg 349f22524e provision: Don’t rely on uid == gid assumption when invoking chown.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Anders Kaseorg 88793e80ea provision: Skip some rm, mkdir, chown commands if possible.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Anders Kaseorg 671844d667 provision: Add default-jre-headless for vnu-jar.
This was previously pulled in by yui-compressor.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-08 17:28:20 -07:00
Tim Abbott de0a41bc9c provision: Fix missing dependency on unzip.
Because this is often installed by default, we hadn't noticed that our
Slack importer doesn't run without it.

Thanks to Ray Kraesig for the report.
2019-08-08 10:49:20 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Wyatt Hoodes f80402c64d tools: Use mypy_extensions for NoReturn.
This is the right way to access it with support for older Python
versions where it might not be in typing yet.
2019-07-31 16:18:24 -07:00
Wyatt Hoodes 927feb615e provision: Comment why NoReturn can't import normally. 2019-07-25 17:48:20 -07:00
Anders Kaseorg 6d5a20ac62 requirements: Remove django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott c2bdccbaa8 provision: Use a shorter threshold for clean-unused-caches.
We've had a few reports of folks running out of disk when we're making
rapid changes to dependencies, and this should help prevent that.
2019-07-24 13:35:00 -07:00
Rishi Gupta 7d8d0b2284 settings: Update upgrade text and styling.
When we add Plus, the first sentence should change to "Available on Zulip
Standard and Plus".

I copied the styling of .tip out of expediency, but it's also possible that
long term we'll want only 1 tip-like box styling.

The hover styling is a bit random, but I tried to copy other hover styles I
found in settings.scss.

Note that this renames .upgrade_realm_plan_type_suggestion to .upgrade-tip.
2019-07-21 14:32:36 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00