Commit Graph

35562 Commits

Author SHA1 Message Date
Steve Howell eb1344c41c list_render: Fix filtering/sorting.
This code has always been kind of convoluted
and buggy, starting with the first
sorting-related commit, which put filtering
before sorting for some reason:

    3706e2c6ba

This should fix bugs like the fact that
changing filter text would not respect
reversed sorts.

Now the scheme is simple:

    - external UI actions set `meta` values like
      filter_value, reverse_mode, and
      sorting_function, as needed, through
      simple setters

    - use `hard_redraw` to do a redraw and
      trigger external actions

    - all filtering/sorting/reverse logic on
      the *data* happens in a single, simple
      function called `filter_and_sort`
2020-04-15 15:13:26 -07:00
Steve Howell ec854f4b12 list_render: Remove opts.filter.event logic.
We don't use this anywhere.  You can do
`git grep -A 40 list_render.create` to verify
this (with a little bit of noise in the grep).

A better strategy for generalizing
this code is to extract the useful logic
into a function that callers can use in their
own custom event handlers, which I'll do
in an upcoming commit.
2020-04-15 15:13:26 -07:00
Steve Howell 3aef11dc0e list_render: Extract get_list_scrolling_container().
We put this in `scroll_util` to make it more likely
we will eventually unify this with other scrolling
logic.  (A big piece to move is ui.get_scroll_element,
but that's for another PR.)

And then the other tactical advantage is that we get
100% line coverage on it.

I changed the warning to an error, since I don't
think we ever expect scrolling at the `body` level,
and I don't bother with the preview node.
2020-04-15 15:13:26 -07:00
Steve Howell 839a817d0e linkifier settings: Simplify default sort.
In 3d7b9a1349 we
removed the `reverse` operation from `sort`, so
this code had bit-rotted, and it's simpler to
just sort by pattern.
2020-04-15 15:13:26 -07:00
shubhamgupta2956 cb71b8c1cf api_docs: Move /rest-error-handling examples.
This commit moves /rest-error-handling examples to components section so
that they can be re-used in individual endpoints where it's example can
be highlighted more easiy.
2020-04-15 12:17:50 -07:00
shubhamgupta2956 f2c9a9a832 api_docs: Add "event_types" common component.
To facilitate re-use of the same parameters in other paths. this commit
store the content of the parameter "event_types" in components.
2020-04-15 12:17:50 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
Mateusz Mandera 5f9da3053d rate_limiter: Handle edge case where rules list may be empty. 2020-04-15 11:20:37 -07:00
Hashir Sarwar b577366a05 rate_limiter: Add an in-process implementation for Tornado.
The Redis-based rate limiting approach takes a lot of time talking to
Redis with 3-4 network requests to Redis on each request.  It had a
negative impact on the performance of `get_events()` since this is our
single highest-traffic endpoint.

This commit introduces an in-process rate limiting alternate for
`/json/events` endpoint. The implementation uses Leaky Bucket
algorithm and Python dictionaries instead of Redis. This drops the
rate limiting time for `get_events()` from about 3000us to less than
100us (on my system).

Fixes #13913.

Co-Author-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
Co-Author-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-15 11:20:37 -07:00
Mateusz Mandera 95fa8b2a26 rate_limiter: Fix too early return if no rules are passed in.
In the redis implementation, if rules was an empty list,
this would return too early - before checking if the key isn't
manually blocked.
2020-04-15 11:20:37 -07:00
Rohitt Vashishtha cb85763c78 settings: Extract DropdownListWidget from default_code_language_widget.
We extract a general purpose widget to create dropdown lists with
search. This widget is used for default code block language, but can
be easily extended to cover notifications_stream and similar options.
The current usage is:

```js
const widget = DropdownListWidget({
    setting_name: 'realm_alpha_beta',
    data: [{name: 'hello', value: 'world'}, {...}, ...],
    subsection: 'msg-editing',
    default_text: 'Nothing is selected',
});
```

and

```handlebars
{{> dropdown_list_widget
  setting_name="realm_alpha_beta"
  list_placeholder=(t 'Filter the data')
  reset_button_text=(t '[Unset]')
  label=admin_settings_label.realm_alpha_beta }}
```

This can further be refined by shifting more variables from handlebars
to javascript in the future.
2020-04-14 12:50:10 -07:00
Rohitt Vashishtha 546c6f99a7 settings: Extract some helper functions from build_page.
By taking these functions out of exports.build_page, we can
reuse them for handling other widgets. We also declare
default_code_language_widget after the helper functions to
avoid the linter complaining.
2020-04-14 12:50:10 -07:00
Puneeth Chaganti b04813d351 webhook/semaphore: Add only summary line to the notification message. 2020-04-14 12:42:36 -07:00
Tim Abbott 0dd0227c8d send_email: Move custom email code to the bottom.
It's of interest to a relatively small subset of developers, in
comparison to Zulip's generic code for sending outgoing emails.
2020-04-14 10:57:20 -07:00
Tim Abbott 2732c477bc send_custom_email: Improve docstrings a bit. 2020-04-14 10:55:04 -07:00
wowol fcf4c9639e emails: Add tests for sending custom emails.
This requires configuring the linter to allow using "subject" in these
files.
2020-04-14 10:50:43 -07:00
wowol 74b757c43c emails: Add support for email headers in send custom email function.
This makes it a bit more convenient to encode most of the email
configuration inside a single template file.
2020-04-14 10:50:29 -07:00
Tim Abbott 8f57ec80f6 help: Add KaTeX links explaining how our LaTeX formatting works.
Also improve the headings for it.

Fixes #14568.
2020-04-13 15:55:39 -07:00
Siddharth Varshney 9cc64156e4 compose: Fix color of preview icon.
Fixes the color of preview iocn to match other message-control-button icons.
2020-04-13 14:58:10 -07:00
Siddharth Varshney ac690a99b8 message-header: Bump up the opcaity of icons.
Because of visibility issues both in day and night mode this commit
will increase the opacity of recipient_bar_controls icons from 0.1 to 0.2.
2020-04-13 14:58:10 -07:00
Steve Howell 4e11e7ee5b Revert "list_render: Clean up initialization."
I pushed this risk commit to the end of
a PR that had a bunch of harmless prep
commits at the front, and I didn't make
it clear enough that the last commit (this
one) hadn't been tested thoroughly.

For the list_render widget, we can simplify
the intialization pretty easily (avoid
extra sorts, for example), but the cache aspects
are still tricky on subsequent calls.
2020-04-13 06:22:28 -04:00
Wowol 0bf5ad3265 emails: Move send custom email function to library. 2020-04-12 16:11:44 -07:00
Ankita Bansal 2b179143a8 static/js: Fix custom numeric emojis not working in reactions.
Changes .data() Jquery methods to .attr() to prevent unnecessary data
type conversions of the emoji name.

Tested the fix manually and verified the test-js-with-node test suite.

Fixes: #14377
2020-04-12 15:02:56 -07:00
Steve Howell 0681e4ba36 list_render: Clean up initialization.
For some widgets we now avoid duplicate redraw
events from this old pattern:

    widget = list_render.create(..., {
    }).init();
    widget.sort(...);

The above code was wasteful and possibly
flicker-y due to the fact that `init` and
`sort` both render.

Now we do this:

    widget = list_render.create(..., {
        init_sort: [...],
    });

For other widgets we just clean up the need
to call `init()` right after `create()`.

We also allow widgets to pass in `sort_fields`
during initialization (since you may want to
have `init_sort` use a custom sort before the
first render.)
2020-04-12 14:59:32 -07:00
Steve Howell ef749dba31 invite settings: Avoid list_render.get() call.
The get() logic here was broken, because
when you enter the settings panel for invites
on the 2nd or 3rd time, the text filter
would not work.

This commit doesn't intend to fix the problem; it
just simplifies the code for a later commit
that fixes this holistically.

The way that we update `list_render` objects
is janky with respect to events, so we can end
up double-sorting lists (which puts them back
to normal) and strange things like that.
2020-04-12 14:59:32 -07:00
Steve Howell 2b07512d22 list_render test: Split out scrolling/filtering.
We split one test into two simpler ones, and we
no longer bother with the load_count override,
which was only used in tests.
2020-04-12 14:59:32 -07:00
Steve Howell ced5511cdd list_render: Rename __set_events().
I rename it to set_up_event_handlers.

This commit does not attempt to fix any buggy
behavior with how we set up event handlers; it's
purely cosmetic.
2020-04-12 14:59:32 -07:00
Steve Howell 42c2e9c429 list_render: Dedent 200+ lines of code.
This is all cosmetic.

Instead of:

    const.widget = {
        foo: function () = {
        },

        bar: function () {
        },
    };

We have:

    const widget = {};

    widget.foo = function () = {
    };

    widget.bar = function () {
    };
2020-04-12 14:59:32 -07:00
Steve Howell 29b22da1ff list_render: Rename prototype to widget. 2020-04-12 14:59:32 -07:00
Steve Howell 22c8a7ccef invite settings: Fix sorting with multi-user links.
Before this fix, we'd get a traceback if you looked
at invites in the settings (and if one of them was
a multi-user invite link).  This commit fixes
that problem by adding a custom sort.

We also rename the "Email" column to "Invitee",
since it's often the case the invitee isn't an
actually an email, but it's instead a multi-use
link.

Note that the invites UI only works the first time you
enter settings.  Many of the controls break the second
time you enter it.  You can't sort by column header
or use the text filter.

I'll fix that in a subsequent commit.
2020-04-12 14:59:32 -07:00
Steve Howell a06d455228 settings: Extract sort helpers for various lists.
Giving these functions a name and moving them to
the top-level scope has a couple tactical advantages:

    - names show in tracebacks
    - code is less indented
    - setup code is less cluttered
    - will be easier to add unit tests
    - will make some upcoming diffs nicer

These are technically more `compare_foo` than `sort_foo`,
but we already had a naming convention that was sort of
in place.
2020-04-12 14:59:32 -07:00
Tim Abbott 4f23f13c55 fetch-contributor-data: Fix running in production.
The previous logic attempted to check for a venv, rather than putting
itself into the virtualenv automatically.
2020-04-12 12:14:07 -07:00
Anders Kaseorg b1e7d8b51d settings: Harden session and CSRF cookies with __Host- prefix.
This defends against cross-origin session fixation attacks.  Renaming
the cookies means this one-time upgrade will have the unfortunate side
effect of logging everyone out, but they’ll get more secure sessions
in return.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-12 11:55:55 -07:00
Anders Kaseorg f47a9408cd settings: Use existing Django setting to mark CSRF cookie HttpOnly.
Instead of sneakily injecting HttpOnly into the cookie via the path
setting, use the setting that was designed for this purpose.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-12 11:55:55 -07:00
Anders Kaseorg dce6b4a40f middleware: Remove unused cookie_domain setting.
Since commit 1d72629dc4, we have been
maintaining a patched copy of Django’s
SessionMiddleware.process_response in order to unconditionally ignore
our own optional cookie_domain setting that we don’t set.

Instead, let’s not do that.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-12 11:55:55 -07:00
Abhishek-Balaji 68257e28ce emoji_name: Raise correct exception if emoji_name is missing.
Right now, the message is "Invalid characters in emoji name" when
the emoji_name is empty. Changing check_valid_emoji_name() in
zerver/lib/emoji.py which validates the name to accomodate the case
of missing name. The new message is "Emoji name is missing".
2020-04-12 11:52:45 -07:00
arpit551 d60efa1478 thumbor: Fix __file__ typo.
Replaced '__file__' typo with __file__ which used to add
wrong path to sys.path.
2020-04-12 11:23:03 -07:00
Puneeth Chaganti 504216c04e webhooks: Remove deprecated Stash webhooks' fixture. 2020-04-12 11:04:30 -07:00
Puneeth Chaganti 67822a0544 messages: Fix 500 on queries with empty string as search operand.
The error is PGroonga specific since `pgroonga_query_extract_keywords` does
not handle empty string inputs correctly. This commit prevents search
narrows from having empty operands.

Closes #14405
2020-04-12 11:04:09 -07:00
Aman Agrawal c792abebc0 docs: Fix broken link to WSL 2 doc. 2020-04-12 11:03:27 -07:00
Aman Agrawal ca0154c425 dev-docs-setup: Add installation instructions for WSL 2.
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.

While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.

Polished by tabbott.
2020-04-11 12:16:29 -07:00
Aman Agrawal cf4349d1c1 tools: Add script to start database services for WSL.
It needs to be run everytime WSL is restarted.
2020-04-11 12:05:33 -07:00
Steve Howell 888214196c unread: Handle message edits that affect mentions.
We had a bug where if your peer mentioned you in
message, but then edited the message not to mention
you, the latter wouldn't reset your unread counts
for "Mentions".  And the same problem would happen
vice versa.

The fix basically extracts `update_message_for_mention`
and makes sure it handles all combinations of
unread/mentioned flags, instead of assuming
any invariants about which directions of change
are possible.

And then we call that new function from
`message_events.js` whenever we get message
edit events.

Fixes #14544
2020-04-11 12:00:46 -07:00
Steve Howell 32157ed6fd node test: Add message for unread/mentions test.
Add the "control" case here of a message that is
already read.
2020-04-11 12:00:46 -07:00
Steve Howell 253a740307 node tests: Use stream message in message_events.
We use a somewhat more realistic message, mostly
to prep for testing some mention/unread stuff in
a subsequent commit.

We also set message booleans.

Unfortunately, `recent_senders` is kind of awkward
for checking a single message, since its only
public API is for sorting.  I don't bother with it.

But I do check the `topic_data` interaction.
2020-04-11 12:00:46 -07:00
Mateusz Mandera 770086f983 url_preview: Discard url in oembed if server returns invalid json.
This fixes the scenario where we'd get errors in the
FetchLinksEmbedData queue processor if oembed got invalid json from the
URL.
2020-04-11 11:54:54 -07:00
Steve Howell b8552e9524 test-js-with-node: Show correct coverage link in color.
We figure out the dev host using the same logic as
dev_settings.py, so that we don't use wrong things
like 127.0.0.1 for droplet users.

And we display the link in cyan.
2020-04-11 07:05:04 -04:00
Steve Howell f28a1a4c6c test-js-with-node: Remove stray comments.
When I introduced EXEMPT_FILES, I forgot
to delete these comments, and they became
totally nonsensical after sorting the list
of files.
2020-04-11 07:05:04 -04:00
wowol 71dfb85607 emails: Allow usage of word subject in send_email.py. 2020-04-10 15:53:35 -07:00
wowol a91ffa7f5f emails: Compile only one template in custom emails. 2020-04-10 15:53:35 -07:00