Commit Graph

35497 Commits

Author SHA1 Message Date
Puneeth Chaganti 464cbcd9c1 tools: Capture incoming bot message screenshot using puppeteer. 2020-04-16 19:25:13 -07:00
Puneeth Chaganti 4d2ce607c9 tools: Add script to trigger webhook notification using fixtures.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.

Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.

Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
2020-04-16 19:25:13 -07:00
Puneeth Chaganti 4c142b778a generate_zulip_bots_static_files: Move imports outside the function. 2020-04-16 19:25:13 -07:00
Tim Abbott 777a3b6c18 puppet: Fix nagios check to not require typing_extensions. 2020-04-16 17:56:05 -07:00
Tim Abbott e1ce53ac46 puppet: Update nagios checks for disk to exclude kernel filesystems.
The fact that we have to explicitly list these is almost certainly a
bug in check_disk, but at least this works.
2020-04-16 17:49:29 -07:00
Tim Abbott cfbb617f5c puppet: Update nagios configuration for checking local disk. 2020-04-16 17:48:36 -07:00
Tim Abbott 9821dfa9fc puppet: The letsencrypt package is debian is now certbot.
It was an alias starting with Ubuntu Xenial, and will eventually be
removed.
2020-04-16 17:30:01 -07:00
Mateusz Mandera 7ed3c3f9f0 saml: Add setting to require limit_to_subdomains on configured IdPs.
If SAML_REQUIRE_LIMIT_TO_SUBDOMAINS is enabled, the configured IdPs will
be validated and cleaned up when the saml backend is initialized.
settings.py would be a tempting and more natural place to do this
perhaps, but in settings.py we don't do logging and we wouldn't be able
to write a test for it.
2020-04-16 17:04:12 -07:00
Mateusz Mandera 143db68422 saml: Implement limiting of IdP to specified realms.
Through the limit_to_subdomains setting on IdP dicts it's now possible
to limit the IdP to only allow authenticating to the specified realms.

Fixes #13340.
2020-04-16 17:04:08 -07:00
Mateusz Mandera 7a9d592dbe tests: Remove out-of-date comment on a saml test for multiple idps. 2020-04-16 17:02:36 -07:00
Tim Abbott 8e5a866122 puppet: Update tuning for load average monitoring. 2020-04-16 16:47:05 -07:00
Tim Abbott 20ac4e1fba docs: Update changelog for Zulip 2.1.4 release. 2020-04-16 16:20:58 -07:00
Tim Abbott b1ff823798 puppet: Remove old zulipbot configuration.
We haven't used zulipbot hosted here for years.
2020-04-16 16:18:48 -07:00
Tim Abbott 1219a2e854 docs: Deprecate support for Xenial and Stretch.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.

Our reasoning for deprecating support for these releases is as follows:

* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
  EOL on the server in about a year.

* Debian Stretch will each EOL in 2020 (the precise date is unclear in
  Debian's documentation, but based on past precedent it's in the next
  few months, perhaps July 2020).
  https://wiki.debian.org/DebianReleases#Production_Releases

* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
  Python, which will reach EOL in September 2020 (and we're already
  seeing various third-party dependencies that we use drop support for
  them).

* While there is LTS support for these older releases, it's not clear it's
  going to be worth the added engineering effort for us to maintain EOL
  releases of the base OSes that we support.

* We (now) have clear upgrade instructions for moving to Debian Buster
  and Ubuntu 18.04.
2020-04-16 15:36:18 -07:00
Pranav daacd2b5bf account settings: Add user_details_section to template.
Added a div(user_details_section) to group all the similar user
settings together into the div. Unindented to make the changes
easily visible.
2020-04-16 15:14:57 -07:00
Pranav e09cccb628 settings-ui: Allow users to view the settings_save_discard_widget.
Removed the condition in the settings_save_discard_widget page which
allowed the page to be rendered only for admins. This change was
important so that this widget can be used on the user settings page
for displaying confirmation messages.

For non-admins, trying to access admin-only settings, behaviour remains
the same as before as these settings are disabled for non-admins.
2020-04-16 15:14:46 -07:00
Tim Abbott d30c8d4a66 terms: Fix email address visibility displayed when accepting ToS.
Apparently, we never tested this unusual flow (primarily used after
Slack import) with an organization with EMAIL_ADDRESS_VISIBILITY_ADMINS.
2020-04-16 11:44:51 -07:00
Abhishek-Balaji b40bd56f3e alert settings: Change style of alert words list.
Narrowing the list of words by reducing margin and padding.
Also reducing the width to 50% of the previous value.
2020-04-15 17:56:14 -07:00
Abhishek-Balaji 52e0063deb alert settings: Move alert word form on top and sort alert words.
The form for entering alert words has been moved above the list
of words.
The list of words will be presented alphabetically rather than
time of addition.
2020-04-15 17:56:14 -07:00
Abhishek-Balaji 255fe13151 alert settings: Include the actual alert word in the success message.
The message now says 'Alert word "foo" added successfully!' when
you add "foo" as an alert word.
2020-04-15 17:56:14 -07:00
Ryan Rehman 9340cd1a0b muting: Send muted_topic's date_muted field to frontend. 2020-04-15 15:48:25 -07:00
Kartik Srivastava a060691389 default streams: Change add api to use stream_id.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.

Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
2020-04-15 15:47:04 -07:00
vaibhavrajsingh2001 2a212b3300 docs: Add documentation for weekly digest emails.
Added documentation explaining the digest emails feature and how to
disable it, along with a link to the /digest.

Resolves: #14136
2020-04-15 15:38:39 -07:00
Tim Abbott 382261dc72 upgrade-zulip: Remove tsearch-extras on upgrade.
We stopped using tsearch-extras in Zulip 2.1.0 after Anders figured
out how to achieve its goals with native postgres.  However, we never
did a `DROP EXTENSION` on systems thta had upgraded, which meant that
backups created on systems originally installed with Zulip 2.0.x and
older, and later upgraded to Zulip 2.1.x, could not be restored on
Zulip servers created with a fresh install of Zulip 2.1.x.

We can't do this with a normal database migration, because DROP
EXTENSION has to be done as the postgres user, so we add some custom
migration code in the upgrade-zulip-stage-2 tool.

It's safe to run this whenever tsearch_extras.control is installed because:
* Zulip is AFAIK the only software that ever used tsearch_extras.
* The package was only installed via puppet on production servers configured to
  run a local Zulip database.
* We'll only run this code once per system, because it removes the
  package and thus the control files.

Fixes #13612.
2020-04-15 15:18:44 -07:00
Jagan 8ae34eaa12 settings UI: Make the settings avatar delete/edit UI slicker.
Added UI support for uploding the new profile picture by
clicking on the avatar rather than a button.
Added new spinner for loading indication while uploading
a new avatar over the avatar area.
Fixes #10255
2020-04-15 15:15:57 -07:00
Steve Howell aa5ffcbd2e admin user: Sort bot owners by name.
The original commit here was sorting bot owners by
id, which is of course meaningless to users:

    444ce74a8e

It was also returning 1/-1 in cases where the bot
owner on both sides of a comparison were missing,
which is a big no-no for sorting algorithms.
2020-04-15 15:13:26 -07:00
Steve Howell 0b71b092b3 admin users: Fix email sort.
The email sort now works correctly for admins in
realms with hidden emails.  (We want to sort on
delivery email.)
2020-04-15 15:13:26 -07:00
Steve Howell cf741e9a11 list_render: Add replace_list_data().
The data() function was used in only one place,
and it can be replaced now with two simple
lines of code.
2020-04-15 15:13:26 -07:00
Steve Howell d406df75b3 list_render: Remove support for jQuery items.
All of our widgets now use templates to turn
items into HTML strings, so we can simplify
the code.
2020-04-15 15:13:26 -07:00
Steve Howell c9d0c6852e admin users: Use plain HTML and static dates.
We want to avoid creating jQuery objects that just
get turned right back into strings by the list
widget, so we now have our template just include
`last_active_date` instead of kludging it in
after the fact, and we return the template
string in `modifier` rather than wrapping it.

To deal with plain HTML we switch to using
`render_now`.

Calling `render_now` leads to a more simple
codepath than `render_date`, beyond just dealing
with text.

The `render_date` function has special-case logic
that only applies to our time dividers in our
message view, which is why we were passing the
strange `undefined` parameter to it before this
fix.

The `render_date` function was also putting
the dates into `update_list` for once-a-day
updates, which is overkill for an admin screen.
We don't use this logic for drafts or attachments
either.  I'm not sure how well tested that logic
is, and it's prone to slow leaks.

This commit sets us up to simplify the list
widget not to have bit-rot-prone code related
to jQuery objects.
2020-04-15 15:13:26 -07:00
sahil839 0d2d7d31e2 admin users: Simplify logic for last active date.
We now:

    - Skip the broken "Never" case.  (The way
      we were distinguishing "Unknown" from
      "Never" was based on brittle checks that
      were just wrong due to bitrot--see Steve
      Shank on czo as an example.  If we want
      to make this distinction rigorous in the
      future, we should have a clear mechanism.
      If somebody's never actually been active,
      we probably want to treat that more like
      a dead-on-arrival login, anyway, and make
      it easy to clean them up.)

    - Use the `presence.last_active_date` instead
      of reaching into private data structures.

    - Avoid the unnecessary intermediate constants
      of LAST_ACTIVE_NEVER and LAST_ACTIVE_UNKOWN.

    - Avoid setting `last_active` in `populate_users`.

This commit was modified by @showell:

    - I cleaned up the commit message.

    - I simplified the diff a bit to avoid
      some renaming and lexical moves.
2020-04-15 15:13:26 -07:00
Steve Howell fad4ad1c7f default streams: Streamline list in settings.
There is no need to wrap these rows in jQuery,
when the list widget is fine with concatenating
strings.
2020-04-15 15:13:26 -07:00
Steve Howell 19cf6d0e04 list_render: Remove data-list-render markup.
We already know which list widget a `<th>`
tag is associated with when we set up the
event handler, so it's silly to read data
from the DOM to find that widget again
when the handler runs.

This commit eliminates a whole class of possible
errors and busy work.
2020-04-15 15:13:26 -07:00
Steve Howell 37eeb90695 list_render: Clean up create/update.
For some widgets we now avoid duplicate redraw
events from this old pattern:

    widget = list_render.create(..., {
    }).init();
    widget.sort(...);

The above code was wasteful and possibly
flicker-y due to the fact that `init` and
`sort` both render.

Now we do this:

    widget = list_render.create(..., {
        init_sort: [...],
    });

For other widgets we just clean up the need
to call `init()` right after `create()`.

We also allow widgets to pass in `sort_fields`
during initialization (since you may want to
have `init_sort` use a custom sort before the
first render.)

Finally, we make the second and third calls
eliminate the prior updates from the previous
widget.  This can prevent strange bugs with
double-reversing columns (although that's
been prevented in a better way with a recent
commit), as well as avoiding double work
with sorting.
2020-04-15 15:13:26 -07:00
Steve Howell 0ff62fc6a2 list_render: Use namespaces for events. 2020-04-15 15:13:26 -07:00
Steve Howell eb1344c41c list_render: Fix filtering/sorting.
This code has always been kind of convoluted
and buggy, starting with the first
sorting-related commit, which put filtering
before sorting for some reason:

    3706e2c6ba

This should fix bugs like the fact that
changing filter text would not respect
reversed sorts.

Now the scheme is simple:

    - external UI actions set `meta` values like
      filter_value, reverse_mode, and
      sorting_function, as needed, through
      simple setters

    - use `hard_redraw` to do a redraw and
      trigger external actions

    - all filtering/sorting/reverse logic on
      the *data* happens in a single, simple
      function called `filter_and_sort`
2020-04-15 15:13:26 -07:00
Steve Howell ec854f4b12 list_render: Remove opts.filter.event logic.
We don't use this anywhere.  You can do
`git grep -A 40 list_render.create` to verify
this (with a little bit of noise in the grep).

A better strategy for generalizing
this code is to extract the useful logic
into a function that callers can use in their
own custom event handlers, which I'll do
in an upcoming commit.
2020-04-15 15:13:26 -07:00
Steve Howell 3aef11dc0e list_render: Extract get_list_scrolling_container().
We put this in `scroll_util` to make it more likely
we will eventually unify this with other scrolling
logic.  (A big piece to move is ui.get_scroll_element,
but that's for another PR.)

And then the other tactical advantage is that we get
100% line coverage on it.

I changed the warning to an error, since I don't
think we ever expect scrolling at the `body` level,
and I don't bother with the preview node.
2020-04-15 15:13:26 -07:00
Steve Howell 839a817d0e linkifier settings: Simplify default sort.
In 3d7b9a1349 we
removed the `reverse` operation from `sort`, so
this code had bit-rotted, and it's simpler to
just sort by pattern.
2020-04-15 15:13:26 -07:00
shubhamgupta2956 cb71b8c1cf api_docs: Move /rest-error-handling examples.
This commit moves /rest-error-handling examples to components section so
that they can be re-used in individual endpoints where it's example can
be highlighted more easiy.
2020-04-15 12:17:50 -07:00
shubhamgupta2956 f2c9a9a832 api_docs: Add "event_types" common component.
To facilitate re-use of the same parameters in other paths. this commit
store the content of the parameter "event_types" in components.
2020-04-15 12:17:50 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
Mateusz Mandera 5f9da3053d rate_limiter: Handle edge case where rules list may be empty. 2020-04-15 11:20:37 -07:00
Hashir Sarwar b577366a05 rate_limiter: Add an in-process implementation for Tornado.
The Redis-based rate limiting approach takes a lot of time talking to
Redis with 3-4 network requests to Redis on each request.  It had a
negative impact on the performance of `get_events()` since this is our
single highest-traffic endpoint.

This commit introduces an in-process rate limiting alternate for
`/json/events` endpoint. The implementation uses Leaky Bucket
algorithm and Python dictionaries instead of Redis. This drops the
rate limiting time for `get_events()` from about 3000us to less than
100us (on my system).

Fixes #13913.

Co-Author-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
Co-Author-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-15 11:20:37 -07:00
Mateusz Mandera 95fa8b2a26 rate_limiter: Fix too early return if no rules are passed in.
In the redis implementation, if rules was an empty list,
this would return too early - before checking if the key isn't
manually blocked.
2020-04-15 11:20:37 -07:00
Rohitt Vashishtha cb85763c78 settings: Extract DropdownListWidget from default_code_language_widget.
We extract a general purpose widget to create dropdown lists with
search. This widget is used for default code block language, but can
be easily extended to cover notifications_stream and similar options.
The current usage is:

```js
const widget = DropdownListWidget({
    setting_name: 'realm_alpha_beta',
    data: [{name: 'hello', value: 'world'}, {...}, ...],
    subsection: 'msg-editing',
    default_text: 'Nothing is selected',
});
```

and

```handlebars
{{> dropdown_list_widget
  setting_name="realm_alpha_beta"
  list_placeholder=(t 'Filter the data')
  reset_button_text=(t '[Unset]')
  label=admin_settings_label.realm_alpha_beta }}
```

This can further be refined by shifting more variables from handlebars
to javascript in the future.
2020-04-14 12:50:10 -07:00
Rohitt Vashishtha 546c6f99a7 settings: Extract some helper functions from build_page.
By taking these functions out of exports.build_page, we can
reuse them for handling other widgets. We also declare
default_code_language_widget after the helper functions to
avoid the linter complaining.
2020-04-14 12:50:10 -07:00
Puneeth Chaganti b04813d351 webhook/semaphore: Add only summary line to the notification message. 2020-04-14 12:42:36 -07:00
Tim Abbott 0dd0227c8d send_email: Move custom email code to the bottom.
It's of interest to a relatively small subset of developers, in
comparison to Zulip's generic code for sending outgoing emails.
2020-04-14 10:57:20 -07:00
Tim Abbott 2732c477bc send_custom_email: Improve docstrings a bit. 2020-04-14 10:55:04 -07:00