Commit Graph

35802 Commits

Author SHA1 Message Date
Mateusz Mandera 3b582931af migration: Migrate messages tied to removed bots.
This cleans up any messages that might have been exchanged with
`NEW_USER_BOT` or `FEEDBACK_BOT` (cross-realm bots that were last
used, as far as we know, years ago) that have been completely removed
from the codebase.

Details on the algorithm are in the migration code itself.

Fixes #13583.
2020-04-26 11:08:39 -07:00
Tim Abbott 9566905b81 docs: Clarify nginx reverse proxy documentation.
Fixes #14740.
2020-04-26 10:51:39 -07:00
rebtung f7fbe3419f api: Improve consistency of reactions API.
Previously, the message and event APIs represented the user differently
for the same reaction data. To make this more consistent, I added a
user_id field to the reaction dict for both messages and events. I
updated the front end to use the user_id field rather than the user
dict. Lastly, I updated front end and back end tests that used user
info.

I primarily tested this by running my local Zulip build and
adding/removing reactions from messages.

Fixes #12049.
2020-04-26 10:35:55 -07:00
Puneeth Chaganti 2a65be2bf5 url preview: Use Chrome's user agent instead of a Zulip one.
Some sites don't render correctly unless you are one of the latest browsers.
YouTube Music, for instance, changes the page title to "Your browser is
deprecated, please upgrade.", which makes our URL previews look bad.
2020-04-26 10:16:43 -07:00
Aman Agrawal 8a841a5634 ci: Update bionic docker images to save upgrade time in builds. 2020-04-25 16:10:55 -07:00
Aman Agrawal cf14cc211f ci: Separate prod tarball extraction and install process.
This allows to run scripts between extraction and install
process.

It will be used to restore npm caches for production install jobs.

We extract the tarball in the working directory so that yarn.lock and
package.json are available to restore cache.

(And also so the path is deterministic).
2020-04-25 16:10:41 -07:00
Aman Agrawal 0aa18e88a5 ci: Stop cloning repo as it is not required for prod install.
Move prod installer and test files to persistent dir so that
they can be utilized.
2020-04-25 16:07:55 -07:00
Aman Agrawal a015160ec0 ci: Fix restoring and and saving npm caches. 2020-04-25 16:07:48 -07:00
Tim Abbott 5e24d26f45 webpack: Remove useless print statement.
In I think every place we run this, you'll already be able to see
`tools/webpack` in the `set -x` style output anyway.
2020-04-25 15:39:22 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Tim Abbott c880886cc3 update-prod-static: Reduce verbosity of logging.
Previously, update-prod-static would produce thousands of lines of
basically useless output in `var/log/update-prod-static.log`.
2020-04-25 15:39:03 -07:00
Anders Kaseorg b46d1c8d07 dev_settings: Fix run-dev SyntaxError.
zthumbor loads dev_settings from Python 2.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 20:08:01 -07:00
pemontto 01a57ddd9d webhooks: Use Gitlab project web_url over repo homepage.
This is preferred because the `repository` field isn't available for
project-level events.
2020-04-24 17:50:47 -07:00
Puneeth Chaganti e41f4b01c0 requirements: Upgrade IPython to the latest version. 2020-04-24 17:25:33 -07:00
Tim Abbott 96443c0a1d settings: Fix missing typing import.
The world previously passed CI only because old versions of
prod_settings_template have this already present via the * import.
2020-04-24 17:08:01 -07:00
Wyatt Hoodes 7f637f7cb1 migrations: Deprecate migration 0077 into a noop.
This ancient migration imports boto, which interferes
with our upgrade to boto3.

> git name-rev f13d6a18eb
f13d6a18eb tags/1.6.0~1082

We can safely assume nobody is upgrading from a server on <1.6.0,
since we have no supported platforms in common with those releases.
2020-04-24 17:03:14 -07:00
Wyatt Hoodes 6d686d0e4b migrations: Deprecate migration 0064 into a noop.
This ancient migration imports boto, which interferes
with our upgrade to boto3.

> git name-rev 8ae35211b5
8ae35211b5 tags/1.6.0~1924

We can safely assume nobody is upgrading from a server on <1.6.0,
since we have no supported platforms in common with those releases.
2020-04-24 17:03:14 -07:00
Wyatt Hoodes 3e8d877564 migrations: Deprecate migration 0060 into a noop.
This ancient migration imports boto, which interferes
with our upgrade to boto3.

> git name-rev a32f666f5c
a32f666f5c tags/1.6.0~2384

We can safely assume nobody is running servers on <1.6.0; there are no
supported platforms in common with 1.6.0 anyway.
2020-04-24 17:03:14 -07:00
Anders Kaseorg c0ffa71fa9 nginx: Replace unanchored regexes in location directives.
We could anchor the regexes, but there’s no need for the power (and
responsibility) of regexes here.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 16:58:19 -07:00
Steve Howell b29c44ed94 list_render test: Test replace_list_data.
This gets us to 100% line coverage for
list_render, so we remove its exemption.
2020-04-24 18:24:12 -04:00
Steve Howell 0365db5ac4 list_render test: Test custom sorting.
In passing, we also get coverage for
list_render.get.

This test also implicitly demonstrates that
you can call `widget.sort` directly from some
external event handler; in other words, you
are not locked into the widget's conventions
for setting up <th> tags.
2020-04-24 18:24:12 -04:00
Steve Howell 47c9f463db list_render test: Split out no-filtering test. 2020-04-24 18:24:12 -04:00
Steve Howell 2cded3f3df list_render test: Check for bad modifier types. 2020-04-24 18:24:12 -04:00
Steve Howell df57f8c006 list_render: Move sort functions to module scope.
Moving these to module scope makes it a little
easier to round out our test coverage, plus
it's a bit less clutter in the widget code.
2020-04-24 18:24:12 -04:00
Steve Howell d9166e2ab1 zblueslip: Simplify broken error message.
We were reporting 1 of 1 when 0 messages were
seen.
2020-04-24 18:24:12 -04:00
Steve Howell 951514dd7d check-templates: Be stricter about singleton tags.
We now forbid tags of the form `<foo ... />` in most
places, and we also forbid it even for several void
tags.

We make exceptions for tags that are already formatted
in two different ways in our codebase.  This is mostly
svg tags, plus these common cases:

    - br
    - hr
    - img
    - input

It would be nice to lock down a convention for these,
even though the HTML specification is unopinionated
on these.  We'll probably want to stay flexible for
svg tags, since they are sometimes copy/pasted from
other sources (although it's probably rare enough for
them that we can tolerate just doing minor edits as
needed).
2020-04-24 20:25:55 +00:00
Steve Howell 28f2a6950e check_templates: Parse '<foo/>' tags better.
If folks put something like '<br/>' in the HTML,
we would think the tag's name was "br/" instead
of "br".  I think we were assuming most folks
would write either "<br>" or <br />".

ASIDE:

We should probably have a consistent
preference among these styles:

    * <br>
    * <br/>
    * <br />

I prefer the first.
2020-04-24 20:25:55 +00:00
Steve Howell ca7c79c93e check-templates: Show filename for tokenize errors. 2020-04-24 20:25:54 +00:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Aman Agrawal ad07814fa4 ci: Run production build and install jobs in sequence. 2020-04-24 12:40:31 -07:00
Aman Agrawal 0e544ec40b ci: Add job to run production script to tests production install.
Only restore npm cache, venv cache not required.
Restart rabbitmq-server for production install.
2020-04-24 12:40:31 -07:00
Aman Agrawal 7c99555107 ci: Copy https headers to persistent dir for comparison. 2020-04-24 12:40:31 -07:00
Aman Agrawal 962af445e9 ci: Add job to build release tarball and save it to be reused.
The tarball will be persistent across jobs in /tmp directory.
We will use this tarball to install production server in a separate
job. This makes sure we are running the production installation
as a normal user would run it and stop any package leaks between
provision and production installation scripts.
2020-04-24 12:40:31 -07:00
Aman Agrawal b7c50034b8 ci: Remove unnecessary setup-production steps.
This steps won't be used since production install will run in
a separate job.
2020-04-24 12:40:26 -07:00
Aman Agrawal 1a096a9a10 ci: Move built production tarball to /tmp/ directory.
This directory will persist across build jobs.
2020-04-24 12:40:20 -07:00
Aman Agrawal 77b24d056a ci: Remove old bionic production build job. 2020-04-24 12:40:10 -07:00
Puneeth Chaganti f2f2080c8b requirements: Unblock IPython upgrade since Python 3.5 dropped. 2020-04-24 12:39:10 -07:00
Puneeth Chaganti 46a4252139 requirements: Add jsx-lexer for syntax highlighting React code. 2020-04-24 12:39:10 -07:00
Pranav 93b09d44d6 ci: Add test to check for untracked files.
Runs a test at the end of tools/ci/backend to check if any untracked
files have been created during this ci tests. Exits with error code 1
if untracked files are found, otherwise exits successfully.

Fixes #14691.
2020-04-24 12:35:47 -07:00
Steve Howell e47e1cd648 droplet settings: Fix hostname-related settings.
We recently changed our droplet setup such that their
host names no longer include zulipdev.org.  This caused
a few things to break.

The particular symptom that this commit fixes is that
we were trying to server static assets from
showell:9991 instead of showell.zulipdev.org:9991,
which meant that you couldn't use the app locally.
(The server would start, but the site's pretty unusable
without static assets.)

Now we rely 100% on `dev_settings.py` to set
`EXTERNAL_HOST` for any droplet users who don't set
that var in their own environment.  That allows us to
remove some essentially duplicate code in `run-dev.py`.

We also set `IS_DEV_DROPLET` explicitly, so that other
code doesn't have to make inferences or duplicate
logic to detemine whether we're a droplet or not.

And then in `settings.py` we use `IS_DEV_DROPLET` to
know that we can use a prod-like method of calculating
`STATIC_URL`, instead of hard coding `localhost`.

We may want to iterate on this further--this was
sort of a quick fix to get droplets functional again.
It's possible we can re-configure droplets to have
folks get reasonable `EXTERNAL_HOST` settings in their
bash profiles, or something like that, although that
may have its own tradeoffs.
2020-04-24 12:33:27 -07:00
Steve Howell 62e9a29064 list_render test: Test reverse sorts.
This also cleans up our `sort_button` helper
a bit.
2020-04-24 10:50:08 -04:00
Steve Howell f2ddd31fe6 list_render: Clean up filter-missing errors. 2020-04-24 10:50:08 -04:00
Steve Howell 059ad86967 list_render: Error if opts are missing.
The check here was too late, and it should
have given a blueslip error.  We obviously
don't expect these errors at runtime; this
is a convenience for developers creating
new widgets.
2020-04-24 10:50:08 -04:00
Steve Howell 5308f892d8 list_render test: Test clear_event_handlers. 2020-04-24 10:50:08 -04:00
Steve Howell 37314cfaee list_render test: Improve filter coverage.
This covers how we wire up the filter, and it
covers using `filterer` instead of `predicate`.

I also go away from the strange length-based
predicate that I had in the original test.
The intention behind the original test was
to show that filters could be more than simple
string-matching, but that was just a strange
way to demonstrate it.
2020-04-24 10:50:08 -04:00
Steve Howell c193751e9b list_render test: Exercise sort handler.
We now make sure that `set_up_event_handlers`
wires up our sort handlers correctly.
2020-04-24 10:50:08 -04:00
Steve Howell 9628cbf922 list_render test: Extract make_scroll_container.
The diff is a little noisy due to renaming a few
poorly named variables, but there aren't any
logic changes here inside the functions.
2020-04-24 10:50:08 -04:00
Steve Howell cc1c517264 node tests: Report coverage url correctly.
This small patch accounts for our new
droplets dropping "zulipdev.org" from
the host name.
2020-04-24 10:50:08 -04:00